Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN
Security + Program Operations
Last Updated: 2026-02-23
Related:
docs/adr/ADR-0039-oshun-studio-resilience-and-error-ux.mddocs/releases/v1/specs/studio-resilience-and-error-ux-dependency-and-data-contract-map.mddocs/releases/v1/specs/studio-resilience-and-error-ux-user-journeys-and-edge-cases.mdapps/oshun/web/src/components/studio/StudioResilienceErrorUxWorkspace.tsxapps/oshun/web/src/analytics/studioResilienceErrorUxTelemetry.ts
Purpose#
Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio resilience and error UX workflows.
This document is the execution artifact for OST-00216.
1. Ownership Model#
| Surface | Primary Owner | Backup Owner | Dependencies |
|---|---|---|---|
| Resilience/error workspace route | OSHUN Web | Platform Web | route contracts + resilience UX orchestration |
| Error envelope normalization | OSHUN BFF | Platform SRE | error APIs + workflow/correlation contract services |
| Recovery/degraded-mode orchestration | Platform SRE | BFF | worker control plane + fallback capability services |
| Severity policy and escalation routing | Security + Program Ops | Compliance | RBAC/policy bundles + incident routing contracts |
| Incident closure/governance packaging | Data Platform + Security | Program Operations | telemetry ingestion + retention/export controls |
2. Core SLOs#
| Metric | Target | Alert Threshold |
|---|---|---|
| Resilience flow success rate | >= 99.0% | < 98.0% over 15 min |
| Error envelope normalization reliability | >= 99.5% | < 99.0% over 15 min |
| Resilience flow p95 latency | <= 2500ms | > 3500ms over 15 min |
| Recovery/degraded transition integrity | >= 99.5% | < 99.0% over 15 min |
| Escalation routing and acknowledgement rate | >= 99.5% | < 99.0% over 15 min |
| Incident closure evidence completeness | >= 99.5% | < 99.0% over 15 min |
3. Operational Playbooks#
3.1 Error Capture and Classification Failures#
Trigger: repeated RESILIENCE_ERROR_CAPTURE_FAILED outcomes.
Actions:
- Validate error envelope identity (
incidentId,workflowId,correlationId) and severity metadata. - Inspect error-normalization adapter output for schema drift.
- Re-run capture in preview with diagnostics enabled.
- Block production promotion until envelope reliability recovers.
3.2 Guided Recovery Failures#
Trigger: repeated RESILIENCE_GUIDED_RECOVERY_FAILED outcomes.
Actions:
- Validate recovery action payload and policy authorization context.
- Inspect recovery state transitions for idempotency drift.
- Re-run recovery flow with controlled blast radius.
- Freeze manual override actions when recovery reliability degrades.
3.3 Degraded-Mode Transition Failures#
Trigger: repeated RESILIENCE_DEGRADED_MODE_TRANSITION_FAILED outcomes.
Actions:
- Validate degraded capability limits and fallback-path metadata.
- Inspect entry/exit transitions for blocked-capability contract mismatches.
- Re-run degraded transition with dependency tracing enabled.
- Escalate to Platform SRE if degraded mode cannot safely stabilize.
3.4 Escalation Routing Failures#
Trigger: repeated RESILIENCE_ESCALATION_ROUTING_FAILED outcomes.
Actions:
- Validate escalation owner routing and severity-policy mapping.
- Inspect notification center and incident-routing service health.
- Re-run escalation with fallback owner routing enabled.
- Trigger manual incident bridge if routing remains unavailable.
3.5 Incident Governance Closure Failures#
Trigger: repeated RESILIENCE_GOVERNANCE_CLOSURE_FAILED outcomes.
Actions:
- Validate closure package root-cause/remediation metadata completeness.
- Inspect governance export schema and retention tagging.
- Re-run closure export with compliance auditor oversight.
- Block production closure until evidence package validation passes.
3.6 Policy Denial Spike#
Trigger: unexpected rise in ROLE_POLICY_DENIED,
CRITICAL_CAPTURE_POLICY_DENIED, MANUAL_OVERRIDE_POLICY_DENIED,
PRODUCTION_DEGRADED_MODE_POLICY_DENIED, or
PRODUCTION_GOVERNANCE_CLOSURE_POLICY_DENIED.
Actions:
- Compare flow allowlists against active RBAC and policy bundles.
- Inspect recent policy release changes for drift.
- Apply policy hotfix or roll back drifted policy bundle.
- Confirm deny-rate normalization and impacted flow recovery.
3.7 Telemetry and Audit Degradation#
Trigger: missing or delayed resilience/error telemetry and audit events.
Actions:
- Validate analytics sink health and ingestion backlog depth.
- Enable degraded-observability warning in resilience workspace.
- Replay buffered events where supported.
- Confirm SLO and evidence stream completeness after recovery.
4. Release Checklist#
- Validate
/studio/resilience-error-uxrenders and executes required flows. - Validate optimistic, error, and retry behavior for each flow.
- Validate critical capture, manual override, degraded mode, and governance policy enforcement.
- Validate telemetry/audit emissions and SLO panel updates.
- Validate integration/e2e/accessibility evidence artifacts.
5. Incident Communication Templates#
Incident Start#
[P{severity}] Studio Resilience/Error Incident
Detected: {timestamp}
Impact: {error-capture | guided-recovery | degraded-mode | escalation-routing | governance-closure}
Suspected cause: {policy drift | adapter regression | degraded transition failure | telemetry degradation}
Owner: {on-call}
Incident Update#
[P{severity}] Studio Resilience/Error Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {captureAndClassifyError | executeGuidedRecovery | enterOrExitDegradedMode | escalateIncidentOwnership | closeIncidentGovernanceExport}
ETA: {time}
Incident Resolved#
[P{severity}] Studio Resilience/Error Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
6. Escalation Paths#
| Area | Primary Channel | Escalate To |
|---|---|---|
| Web runtime and resilience UX | #oshun-web |
Platform on-call |
| Error normalization and BFF APIs | #oshun-bff |
BFF lead |
| Recovery/degraded orchestration | #oshun-platform-sre |
SRE lead |
| Severity policy/escalation rules | #oshun-security |
Security lead |
| Governance closure evidence paths | #oshun-data-platform |
Data platform/SRE lead |
7. Acceptance Criteria (OST-00216)#
OST-00216 is complete only when:
- Ownership and decision authority are documented.
- Resilience/error SLOs and alert thresholds are documented.
- At least five playbooks are documented for major failure modes.
- Release checklist, communication templates, and escalation paths exist.