# Isis Remediation Risk Board (Execution, Infra, Cost)

Date: 2026-02-21  
Source: `ISIS_TODOS.md`, `docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md`, `docs/domains/isis/remediation/ISIS_MILESTONES.md`, `docs/domains/isis/remediation/ISIS_FINDING_OWNERSHIP.md`

## Purpose
Track and control the highest-probability and highest-impact regressions during Isis remediation delivery, with explicit ownership and ticket-linked mitigation plans.

## Scoring Model
- **Likelihood**: Low / Medium / High
- **Impact**: Medium / High / Critical
- **Exposure**: `P0` (High/Critical), `P1` (Medium), `P2` (Low)

## Risk Board

| Risk ID | Category | Risk Statement | Likelihood | Impact | Exposure | Owner | Leading Indicators | Mitigation Plan | Contingency Trigger | Linked Work |
|---|---|---|---|---|---|---|---|---|---|---|
| R-EX-001 | Execution | Jobs are accepted but do not execute due to queue/worker path regressions. | High | Critical | P0 | `@oshun/isis-team` | Queue depth grows while `running` rate is flat; `resolve_failure` spikes; no worker heartbeats | Complete queue producer + lifecycle wiring + DLQ (`F01-*`), enforce startup mapping validation (`F02-*`), add queue health metrics (`F17-*`) | `queued > 200` for 10m or `no running jobs` for 5m during load test | `F01`, `F02`, `F17`, `GATE-001`, `GATE-002` |
| R-EX-002 | Execution | Workflow schema mismatch causes runtime rejection or silent feature dropout. | High | High | P0 | `@oshun/isis-team` | Schema validation failure rate > 1%; registry import rejects canonical workflows; rise in fallback usage | Canonical schema ADR + compatibility layer + strict validator tests (`F03-*`, `F14-*`, `F15-*`) | Any release candidate with unresolved `workflow_schema_compat` failures | `F03`, `F14`, `F15`, `GATE-001` |
| R-EX-003 | Execution | Simulated executor paths leak into production profile and mask failed integrations. | Medium | High | P0 | `@oshun/ml-team` | Placeholder URL output appears in prod logs; zero provider call volume with non-zero completion count | Remove simulation flags from prod configs, enforce fail-closed runtime checks (`F07-*`, `F11-*`, `OPS-*`) | Detection of placeholder artifact in production telemetry | `F07`, `F11`, `OPS-001`, `OPS-008` |
| R-INF-001 | Infrastructure | RunPod endpoint provisioning drifts from expected handler map and breaks routing. | High | High | P0 | `@oshun/devops` | Drift report non-empty; endpoint health mismatch; handler-to-workflow mapping gaps | Provisioning automation + drift detection + fail CI on unmanaged changes (`F08-*`, `F05-*`, `F06-*`) | Any unmanaged endpoint delta in staging/prod | `F05`, `F06`, `F08`, `GATE-003` |
| R-INF-002 | Infrastructure | Readiness signals stay green despite provider/storage/queue degradation. | Medium | Critical | P0 | `@oshun/platform` | `/readyz` green while dependent checks fail; SLO burn without alert | Truthful readiness decomposition + dependency-aware probes and alerts (`F11-*`, `F17-*`, `OPS-*`) | 2 consecutive synthetic failures with green readiness | `F11`, `F17`, `OPS-006`, `OPS-010` |
| R-INF-003 | Infrastructure | Deployment config/path drift causes missing routes or broken web app surfaces. | Medium | High | P1 | `@oshun/devops` | Compose/K8s manifests diverge from app routes; static asset 404 increases | Standardize manifests and environment contracts, add startup asserts and smoke tests (`F13-*`, `F04-*`, `F14-*`) | Any failed launch smoke test on mandatory route matrix | `F13`, `F04`, `F14`, `UX-*` |
| R-COST-001 | Cost | Missing runtime-seconds and pricing fields prevents true pay-per-second accounting. | High | High | P0 | `@oshun/platform` | Job history lacks `runtimeSeconds` or `costUsd`; cost deltas cannot be reconciled | Implement authoritative cost ledger + persistence + reconciliation tests (`F05-*`, `F17-*`, `F18-*`) | More than 0.5% jobs missing billable fields in any day | `F05`, `F17`, `F18`, `GATE-004` |
| R-COST-002 | Cost | Retry storms and uncapped workflows create runaway RunPod spend. | Medium | Critical | P0 | `@oshun/ml-team` | Retry rate > threshold; budget burn > 20% over forecast; runaway execution duration tails | Add retry caps/backoff/circuit breakers + per-workflow guardrails (`F01-*`, `F06-*`, `F17-*`) | Forecasted weekly spend breach > 15% for 2 consecutive days | `F01`, `F06`, `F17`, `OPS-004` |
| R-COST-003 | Cost | Output storage and artifact retention policies create uncontrolled cost growth. | Medium | Medium | P1 | `@oshun/isis-team` | Storage growth exceeds planned slope; lifecycle jobs lag | Enforce TTL/lifecycle and ownership controls for outputs and model assets (`F10-*`, `F12-*`, `F18-*`) | Storage trend exceeds baseline by > 25% week-over-week | `F10`, `F12`, `F18`, `OPS-011` |

## Operating Cadence
- Weekly risk review every Monday during remediation standup.
- Any `P0` risk in "triggered" state must be assigned an incident owner within 30 minutes.
- No milestone gate can close while any `P0` risk remains unmitigated without written exception from `@oshun/core-team`.

## Escalation Rules
- Escalate to `@oshun/core-team` when a contingency trigger is met.
- Freeze release promotion when any of:
  - `R-EX-001`, `R-INF-001`, or `R-COST-001` is triggered
  - Gate validation (`GATE-*`) fails for the candidate milestone
- Require security review for risks intersecting tenant boundaries (`R-INF-002`, `R-COST-003`).

## Reporting Snapshot Template
Use this block in weekly updates:

```md
### Isis Risk Snapshot (YYYY-MM-DD)
- Triggered risks:
- Newly mitigated risks:
- Exposure changes (P0/P1/P2):
- Required decisions/escalations:
```

