Eve library

OSHUN Studio Resilience and Error UX Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0039-oshun-studio-resilience-and-error-ux.md
  • docs/releases/v1/specs/studio-resilience-and-error-ux-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-resilience-and-error-ux-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioResilienceErrorUxWorkspace.tsx
  • apps/oshun/web/src/analytics/studioResilienceErrorUxTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio resilience and error UX workflows.

This document is the execution artifact for OST-00216.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Resilience/error workspace route OSHUN Web Platform Web route contracts + resilience UX orchestration
Error envelope normalization OSHUN BFF Platform SRE error APIs + workflow/correlation contract services
Recovery/degraded-mode orchestration Platform SRE BFF worker control plane + fallback capability services
Severity policy and escalation routing Security + Program Ops Compliance RBAC/policy bundles + incident routing contracts
Incident closure/governance packaging Data Platform + Security Program Operations telemetry ingestion + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
Resilience flow success rate >= 99.0% < 98.0% over 15 min
Error envelope normalization reliability >= 99.5% < 99.0% over 15 min
Resilience flow p95 latency <= 2500ms > 3500ms over 15 min
Recovery/degraded transition integrity >= 99.5% < 99.0% over 15 min
Escalation routing and acknowledgement rate >= 99.5% < 99.0% over 15 min
Incident closure evidence completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Error Capture and Classification Failures#

Trigger: repeated RESILIENCE_ERROR_CAPTURE_FAILED outcomes.

Actions:

  1. Validate error envelope identity (incidentId, workflowId, correlationId) and severity metadata.
  2. Inspect error-normalization adapter output for schema drift.
  3. Re-run capture in preview with diagnostics enabled.
  4. Block production promotion until envelope reliability recovers.

3.2 Guided Recovery Failures#

Trigger: repeated RESILIENCE_GUIDED_RECOVERY_FAILED outcomes.

Actions:

  1. Validate recovery action payload and policy authorization context.
  2. Inspect recovery state transitions for idempotency drift.
  3. Re-run recovery flow with controlled blast radius.
  4. Freeze manual override actions when recovery reliability degrades.

3.3 Degraded-Mode Transition Failures#

Trigger: repeated RESILIENCE_DEGRADED_MODE_TRANSITION_FAILED outcomes.

Actions:

  1. Validate degraded capability limits and fallback-path metadata.
  2. Inspect entry/exit transitions for blocked-capability contract mismatches.
  3. Re-run degraded transition with dependency tracing enabled.
  4. Escalate to Platform SRE if degraded mode cannot safely stabilize.

3.4 Escalation Routing Failures#

Trigger: repeated RESILIENCE_ESCALATION_ROUTING_FAILED outcomes.

Actions:

  1. Validate escalation owner routing and severity-policy mapping.
  2. Inspect notification center and incident-routing service health.
  3. Re-run escalation with fallback owner routing enabled.
  4. Trigger manual incident bridge if routing remains unavailable.

3.5 Incident Governance Closure Failures#

Trigger: repeated RESILIENCE_GOVERNANCE_CLOSURE_FAILED outcomes.

Actions:

  1. Validate closure package root-cause/remediation metadata completeness.
  2. Inspect governance export schema and retention tagging.
  3. Re-run closure export with compliance auditor oversight.
  4. Block production closure until evidence package validation passes.

3.6 Policy Denial Spike#

Trigger: unexpected rise in ROLE_POLICY_DENIED, CRITICAL_CAPTURE_POLICY_DENIED, MANUAL_OVERRIDE_POLICY_DENIED, PRODUCTION_DEGRADED_MODE_POLICY_DENIED, or PRODUCTION_GOVERNANCE_CLOSURE_POLICY_DENIED.

Actions:

  1. Compare flow allowlists against active RBAC and policy bundles.
  2. Inspect recent policy release changes for drift.
  3. Apply policy hotfix or roll back drifted policy bundle.
  4. Confirm deny-rate normalization and impacted flow recovery.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed resilience/error telemetry and audit events.

Actions:

  1. Validate analytics sink health and ingestion backlog depth.
  2. Enable degraded-observability warning in resilience workspace.
  3. Replay buffered events where supported.
  4. Confirm SLO and evidence stream completeness after recovery.

4. Release Checklist#

  • Validate /studio/resilience-error-ux renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate critical capture, manual override, degraded mode, and governance policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Resilience/Error Incident
Detected: {timestamp}
Impact: {error-capture | guided-recovery | degraded-mode | escalation-routing | governance-closure}
Suspected cause: {policy drift | adapter regression | degraded transition failure | telemetry degradation}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Resilience/Error Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {captureAndClassifyError | executeGuidedRecovery | enterOrExitDegradedMode | escalateIncidentOwnership | closeIncidentGovernanceExport}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Resilience/Error Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and resilience UX #oshun-web Platform on-call
Error normalization and BFF APIs #oshun-bff BFF lead
Recovery/degraded orchestration #oshun-platform-sre SRE lead
Severity policy/escalation rules #oshun-security Security lead
Governance closure evidence paths #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00216)#

OST-00216 is complete only when:

  1. Ownership and decision authority are documented.
  2. Resilience/error SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.