Date: 2026-02-21
Source: ISIS_TODOS.md, docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md, docs/domains/isis/remediation/ISIS_MILESTONES.md, docs/domains/isis/remediation/ISIS_FINDING_OWNERSHIP.md
Purpose#
Track and control the highest-probability and highest-impact regressions during Isis remediation delivery, with explicit ownership and ticket-linked mitigation plans.
Scoring Model#
- Likelihood: Low / Medium / High
- Impact: Medium / High / Critical
- Exposure:
P0(High/Critical),P1(Medium),P2(Low)
Risk Board#
| Risk ID | Category | Risk Statement | Likelihood | Impact | Exposure | Owner | Leading Indicators | Mitigation Plan | Contingency Trigger | Linked Work |
|---|---|---|---|---|---|---|---|---|---|---|
| R-EX-001 | Execution | Jobs are accepted but do not execute due to queue/worker path regressions. | High | Critical | P0 | @oshun/isis-team |
Queue depth grows while running rate is flat; resolve_failure spikes; no worker heartbeats |
Complete queue producer + lifecycle wiring + DLQ (F01-*), enforce startup mapping validation (F02-*), add queue health metrics (F17-*) |
queued > 200 for 10m or no running jobs for 5m during load test |
F01, F02, F17, GATE-001, GATE-002 |
| R-EX-002 | Execution | Workflow schema mismatch causes runtime rejection or silent feature dropout. | High | High | P0 | @oshun/isis-team |
Schema validation failure rate > 1%; registry import rejects canonical workflows; rise in fallback usage | Canonical schema ADR + compatibility layer + strict validator tests (F03-*, F14-*, F15-*) |
Any release candidate with unresolved workflow_schema_compat failures |
F03, F14, F15, GATE-001 |
| R-EX-003 | Execution | Simulated executor paths leak into production profile and mask failed integrations. | Medium | High | P0 | @oshun/ml-team |
Placeholder URL output appears in prod logs; zero provider call volume with non-zero completion count | Remove simulation flags from prod configs, enforce fail-closed runtime checks (F07-*, F11-*, OPS-*) |
Detection of placeholder artifact in production telemetry | F07, F11, OPS-001, OPS-008 |
| R-INF-001 | Infrastructure | RunPod endpoint provisioning drifts from expected handler map and breaks routing. | High | High | P0 | @oshun/devops |
Drift report non-empty; endpoint health mismatch; handler-to-workflow mapping gaps | Provisioning automation + drift detection + fail CI on unmanaged changes (F08-*, F05-*, F06-*) |
Any unmanaged endpoint delta in staging/prod | F05, F06, F08, GATE-003 |
| R-INF-002 | Infrastructure | Readiness signals stay green despite provider/storage/queue degradation. | Medium | Critical | P0 | @oshun/platform |
/readyz green while dependent checks fail; SLO burn without alert |
Truthful readiness decomposition + dependency-aware probes and alerts (F11-*, F17-*, OPS-*) |
2 consecutive synthetic failures with green readiness | F11, F17, OPS-006, OPS-010 |
| R-INF-003 | Infrastructure | Deployment config/path drift causes missing routes or broken web app surfaces. | Medium | High | P1 | @oshun/devops |
Compose/K8s manifests diverge from app routes; static asset 404 increases | Standardize manifests and environment contracts, add startup asserts and smoke tests (F13-*, F04-*, F14-*) |
Any failed launch smoke test on mandatory route matrix | F13, F04, F14, UX-* |
| R-COST-001 | Cost | Missing runtime-seconds and pricing fields prevents true pay-per-second accounting. | High | High | P0 | @oshun/platform |
Job history lacks runtimeSeconds or costUsd; cost deltas cannot be reconciled |
Implement authoritative cost ledger + persistence + reconciliation tests (F05-*, F17-*, F18-*) |
More than 0.5% jobs missing billable fields in any day | F05, F17, F18, GATE-004 |
| R-COST-002 | Cost | Retry storms and uncapped workflows create runaway RunPod spend. | Medium | Critical | P0 | @oshun/ml-team |
Retry rate > threshold; budget burn > 20% over forecast; runaway execution duration tails | Add retry caps/backoff/circuit breakers + per-workflow guardrails (F01-*, F06-*, F17-*) |
Forecasted weekly spend breach > 15% for 2 consecutive days | F01, F06, F17, OPS-004 |
| R-COST-003 | Cost | Output storage and artifact retention policies create uncontrolled cost growth. | Medium | Medium | P1 | @oshun/isis-team |
Storage growth exceeds planned slope; lifecycle jobs lag | Enforce TTL/lifecycle and ownership controls for outputs and model assets (F10-*, F12-*, F18-*) |
Storage trend exceeds baseline by > 25% week-over-week | F10, F12, F18, OPS-011 |
Operating Cadence#
- Weekly risk review every Monday during remediation standup.
- Any
P0risk in "triggered" state must be assigned an incident owner within 30 minutes. - No milestone gate can close while any
P0risk remains unmitigated without written exception from@oshun/core-team.
Escalation Rules#
- Escalate to
@oshun/core-teamwhen a contingency trigger is met. - Freeze release promotion when any of:
R-EX-001,R-INF-001, orR-COST-001is triggered- Gate validation (
GATE-*) fails for the candidate milestone
- Require security review for risks intersecting tenant boundaries (
R-INF-002,R-COST-003).
Reporting Snapshot Template#
Use this block in weekly updates:
md
### Isis Risk Snapshot (YYYY-MM-DD)
- Triggered risks:
- Newly mitigated risks:
- Exposure changes (P0/P1/P2):
- Required decisions/escalations: