# OSHUN Studio Resilience and Error UX Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN
Security + Program Operations  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0039-oshun-studio-resilience-and-error-ux.md`
- `docs/releases/v1/specs/studio-resilience-and-error-ux-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-resilience-and-error-ux-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioResilienceErrorUxWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioResilienceErrorUxTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio resilience and error UX workflows.

This document is the execution artifact for `OST-00216`.

## 1. Ownership Model

| Surface                                | Primary Owner            | Backup Owner       | Dependencies                                        |
| -------------------------------------- | ------------------------ | ------------------ | --------------------------------------------------- |
| Resilience/error workspace route       | OSHUN Web                | Platform Web       | route contracts + resilience UX orchestration       |
| Error envelope normalization           | OSHUN BFF                | Platform SRE       | error APIs + workflow/correlation contract services |
| Recovery/degraded-mode orchestration   | Platform SRE             | BFF                | worker control plane + fallback capability services |
| Severity policy and escalation routing | Security + Program Ops   | Compliance         | RBAC/policy bundles + incident routing contracts    |
| Incident closure/governance packaging  | Data Platform + Security | Program Operations | telemetry ingestion + retention/export controls     |

## 2. Core SLOs

| Metric                                      | Target    | Alert Threshold      |
| ------------------------------------------- | --------- | -------------------- |
| Resilience flow success rate                | >= 99.0%  | < 98.0% over 15 min  |
| Error envelope normalization reliability    | >= 99.5%  | < 99.0% over 15 min  |
| Resilience flow p95 latency                 | <= 2500ms | > 3500ms over 15 min |
| Recovery/degraded transition integrity      | >= 99.5%  | < 99.0% over 15 min  |
| Escalation routing and acknowledgement rate | >= 99.5%  | < 99.0% over 15 min  |
| Incident closure evidence completeness      | >= 99.5%  | < 99.0% over 15 min  |

## 3. Operational Playbooks

### 3.1 Error Capture and Classification Failures

Trigger: repeated `RESILIENCE_ERROR_CAPTURE_FAILED` outcomes.

Actions:

1. Validate error envelope identity (`incidentId`, `workflowId`,
   `correlationId`) and severity metadata.
2. Inspect error-normalization adapter output for schema drift.
3. Re-run capture in preview with diagnostics enabled.
4. Block production promotion until envelope reliability recovers.

### 3.2 Guided Recovery Failures

Trigger: repeated `RESILIENCE_GUIDED_RECOVERY_FAILED` outcomes.

Actions:

1. Validate recovery action payload and policy authorization context.
2. Inspect recovery state transitions for idempotency drift.
3. Re-run recovery flow with controlled blast radius.
4. Freeze manual override actions when recovery reliability degrades.

### 3.3 Degraded-Mode Transition Failures

Trigger: repeated `RESILIENCE_DEGRADED_MODE_TRANSITION_FAILED` outcomes.

Actions:

1. Validate degraded capability limits and fallback-path metadata.
2. Inspect entry/exit transitions for blocked-capability contract mismatches.
3. Re-run degraded transition with dependency tracing enabled.
4. Escalate to Platform SRE if degraded mode cannot safely stabilize.

### 3.4 Escalation Routing Failures

Trigger: repeated `RESILIENCE_ESCALATION_ROUTING_FAILED` outcomes.

Actions:

1. Validate escalation owner routing and severity-policy mapping.
2. Inspect notification center and incident-routing service health.
3. Re-run escalation with fallback owner routing enabled.
4. Trigger manual incident bridge if routing remains unavailable.

### 3.5 Incident Governance Closure Failures

Trigger: repeated `RESILIENCE_GOVERNANCE_CLOSURE_FAILED` outcomes.

Actions:

1. Validate closure package root-cause/remediation metadata completeness.
2. Inspect governance export schema and retention tagging.
3. Re-run closure export with compliance auditor oversight.
4. Block production closure until evidence package validation passes.

### 3.6 Policy Denial Spike

Trigger: unexpected rise in `ROLE_POLICY_DENIED`,
`CRITICAL_CAPTURE_POLICY_DENIED`, `MANUAL_OVERRIDE_POLICY_DENIED`,
`PRODUCTION_DEGRADED_MODE_POLICY_DENIED`, or
`PRODUCTION_GOVERNANCE_CLOSURE_POLICY_DENIED`.

Actions:

1. Compare flow allowlists against active RBAC and policy bundles.
2. Inspect recent policy release changes for drift.
3. Apply policy hotfix or roll back drifted policy bundle.
4. Confirm deny-rate normalization and impacted flow recovery.

### 3.7 Telemetry and Audit Degradation

Trigger: missing or delayed resilience/error telemetry and audit events.

Actions:

1. Validate analytics sink health and ingestion backlog depth.
2. Enable degraded-observability warning in resilience workspace.
3. Replay buffered events where supported.
4. Confirm SLO and evidence stream completeness after recovery.

## 4. Release Checklist

- [ ] Validate `/studio/resilience-error-ux` renders and executes required
      flows.
- [ ] Validate optimistic, error, and retry behavior for each flow.
- [ ] Validate critical capture, manual override, degraded mode, and governance
      policy enforcement.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```
[P{severity}] Studio Resilience/Error Incident
Detected: {timestamp}
Impact: {error-capture | guided-recovery | degraded-mode | escalation-routing | governance-closure}
Suspected cause: {policy drift | adapter regression | degraded transition failure | telemetry degradation}
Owner: {on-call}
```

### Incident Update

```
[P{severity}] Studio Resilience/Error Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {captureAndClassifyError | executeGuidedRecovery | enterOrExitDegradedMode | escalateIncidentOwnership | closeIncidentGovernanceExport}
ETA: {time}
```

### Incident Resolved

```
[P{severity}] Studio Resilience/Error Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                              | Primary Channel        | Escalate To            |
| --------------------------------- | ---------------------- | ---------------------- |
| Web runtime and resilience UX     | `#oshun-web`           | Platform on-call       |
| Error normalization and BFF APIs  | `#oshun-bff`           | BFF lead               |
| Recovery/degraded orchestration   | `#oshun-platform-sre`  | SRE lead               |
| Severity policy/escalation rules  | `#oshun-security`      | Security lead          |
| Governance closure evidence paths | `#oshun-data-platform` | Data platform/SRE lead |

## 7. Acceptance Criteria (OST-00216)

`OST-00216` is complete only when:

1. Ownership and decision authority are documented.
2. Resilience/error SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
