Eve library

Isis Remediation Risk Board (Execution, Infra, Cost)

Guide

Revision 1

On this page

Date: 2026-02-21
Source: ISIS_TODOS.md, docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md, docs/domains/isis/remediation/ISIS_MILESTONES.md, docs/domains/isis/remediation/ISIS_FINDING_OWNERSHIP.md

Purpose#

Track and control the highest-probability and highest-impact regressions during Isis remediation delivery, with explicit ownership and ticket-linked mitigation plans.

Scoring Model#

  • Likelihood: Low / Medium / High
  • Impact: Medium / High / Critical
  • Exposure: P0 (High/Critical), P1 (Medium), P2 (Low)

Risk Board#

Risk ID Category Risk Statement Likelihood Impact Exposure Owner Leading Indicators Mitigation Plan Contingency Trigger Linked Work
R-EX-001 Execution Jobs are accepted but do not execute due to queue/worker path regressions. High Critical P0 @oshun/isis-team Queue depth grows while running rate is flat; resolve_failure spikes; no worker heartbeats Complete queue producer + lifecycle wiring + DLQ (F01-*), enforce startup mapping validation (F02-*), add queue health metrics (F17-*) queued > 200 for 10m or no running jobs for 5m during load test F01, F02, F17, GATE-001, GATE-002
R-EX-002 Execution Workflow schema mismatch causes runtime rejection or silent feature dropout. High High P0 @oshun/isis-team Schema validation failure rate > 1%; registry import rejects canonical workflows; rise in fallback usage Canonical schema ADR + compatibility layer + strict validator tests (F03-*, F14-*, F15-*) Any release candidate with unresolved workflow_schema_compat failures F03, F14, F15, GATE-001
R-EX-003 Execution Simulated executor paths leak into production profile and mask failed integrations. Medium High P0 @oshun/ml-team Placeholder URL output appears in prod logs; zero provider call volume with non-zero completion count Remove simulation flags from prod configs, enforce fail-closed runtime checks (F07-*, F11-*, OPS-*) Detection of placeholder artifact in production telemetry F07, F11, OPS-001, OPS-008
R-INF-001 Infrastructure RunPod endpoint provisioning drifts from expected handler map and breaks routing. High High P0 @oshun/devops Drift report non-empty; endpoint health mismatch; handler-to-workflow mapping gaps Provisioning automation + drift detection + fail CI on unmanaged changes (F08-*, F05-*, F06-*) Any unmanaged endpoint delta in staging/prod F05, F06, F08, GATE-003
R-INF-002 Infrastructure Readiness signals stay green despite provider/storage/queue degradation. Medium Critical P0 @oshun/platform /readyz green while dependent checks fail; SLO burn without alert Truthful readiness decomposition + dependency-aware probes and alerts (F11-*, F17-*, OPS-*) 2 consecutive synthetic failures with green readiness F11, F17, OPS-006, OPS-010
R-INF-003 Infrastructure Deployment config/path drift causes missing routes or broken web app surfaces. Medium High P1 @oshun/devops Compose/K8s manifests diverge from app routes; static asset 404 increases Standardize manifests and environment contracts, add startup asserts and smoke tests (F13-*, F04-*, F14-*) Any failed launch smoke test on mandatory route matrix F13, F04, F14, UX-*
R-COST-001 Cost Missing runtime-seconds and pricing fields prevents true pay-per-second accounting. High High P0 @oshun/platform Job history lacks runtimeSeconds or costUsd; cost deltas cannot be reconciled Implement authoritative cost ledger + persistence + reconciliation tests (F05-*, F17-*, F18-*) More than 0.5% jobs missing billable fields in any day F05, F17, F18, GATE-004
R-COST-002 Cost Retry storms and uncapped workflows create runaway RunPod spend. Medium Critical P0 @oshun/ml-team Retry rate > threshold; budget burn > 20% over forecast; runaway execution duration tails Add retry caps/backoff/circuit breakers + per-workflow guardrails (F01-*, F06-*, F17-*) Forecasted weekly spend breach > 15% for 2 consecutive days F01, F06, F17, OPS-004
R-COST-003 Cost Output storage and artifact retention policies create uncontrolled cost growth. Medium Medium P1 @oshun/isis-team Storage growth exceeds planned slope; lifecycle jobs lag Enforce TTL/lifecycle and ownership controls for outputs and model assets (F10-*, F12-*, F18-*) Storage trend exceeds baseline by > 25% week-over-week F10, F12, F18, OPS-011

Operating Cadence#

  • Weekly risk review every Monday during remediation standup.
  • Any P0 risk in "triggered" state must be assigned an incident owner within 30 minutes.
  • No milestone gate can close while any P0 risk remains unmitigated without written exception from @oshun/core-team.

Escalation Rules#

  • Escalate to @oshun/core-team when a contingency trigger is met.
  • Freeze release promotion when any of:
    • R-EX-001, R-INF-001, or R-COST-001 is triggered
    • Gate validation (GATE-*) fails for the candidate milestone
  • Require security review for risks intersecting tenant boundaries (R-INF-002, R-COST-003).

Reporting Snapshot Template#

Use this block in weekly updates:

md
### Isis Risk Snapshot (YYYY-MM-DD)
- Triggered risks:
- Newly mitigated risks:
- Exposure changes (P0/P1/P2):
- Required decisions/escalations: