Eve library

OSHUN Studio Backup and Disaster Recovery UX Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Data Platform + Platform SRE + OSHUN Security and Compliance + OSHUN Web + OSHUN BFF + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0051-oshun-studio-backup-and-disaster-recovery-ux.md
  • docs/releases/v1/specs/studio-backup-and-disaster-recovery-ux-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-backup-and-disaster-recovery-ux-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioBackupDisasterRecoveryUxWorkspace.tsx
  • apps/oshun/web/src/analytics/studioBackupDisasterRecoveryUxTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio backup and disaster recovery workflows.

This document is the execution artifact for OST-00312.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Backup/disaster recovery workspace route OSHUN Web Platform Web workflow UX orchestration + incident navigation
Backup plan registry and schedule governance Data Platform + SRE Studio Architecture backup metadata schemas + retention controls
Restore simulation and integrity evaluation Platform SRE + Data Platform Security restore-point integrity signals + execution telemetry
Failover/restore execution governance Platform SRE + Compliance Program Operations role/tier enforcement + CAB policy + waiver lifecycle
Recovery verification and incident resolution Platform SRE + Security Data Platform verification evidence + incident-state orchestration
Telemetry and governance audit continuity Data Platform + Security Platform SRE analytics sinks + retention/export + replay semantics

2. Core SLOs#

Metric Target Alert Threshold
Backup/DR flow success rate >= 99.0% < 98.0% over 15 min
Backup plan publish and schedule integrity >= 99.5% < 99.0% over 15 min
Simulation/execute/verify flow p95 <= 2500ms > 3500ms over 15 min
Restore-point integrity gate correctness >= 99.5% < 99.0% over 15 min
Policy-tier/CAB/waiver enforcement accuracy >= 99.5% < 99.0% over 15 min
Recovery verification and closure determinism >= 99.5% < 99.0% over 15 min
Telemetry/audit event delivery continuity >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Backup Plan Baseline Failures#

Trigger: repeated BACKUP_DR_PLAN_BASELINE_FAILED outcomes.

Actions:

  1. Validate plan metadata (planId, scope, schedule, retentionDays, encryptionProfile).
  2. Inspect schedule parser behavior and retention policy boundary enforcement.
  3. Re-run baseline flow in preview with diagnostics and policy snapshots.
  4. Block downstream execute flows for plans with invalid baseline state.

3.2 Restore Simulation and Integrity Failures#

Trigger: repeated BACKUP_DR_SIMULATION_FAILED outcomes.

Actions:

  1. Validate restore mode and restore-point integrity (restorePointId, integrityState, coveragePercent).
  2. Inspect restore-point ingestion delay, checksum drift, and storage-tier hydration status.
  3. Re-run simulation with known-good restore point and bounded blast radius.
  4. Hold execute-mode promotion until integrity verification recovers.

3.3 Failover/Restore Execution Policy Denials#

Trigger: repeated BACKUP_DR_EXECUTION_POLICY_DENIED outcomes.

Actions:

  1. Validate role/tier/waiver/CAB prerequisites (role, permissionTier, waiverReason, approvalTicket, releaseChannel).
  2. Inspect policy bundle version drift and approver assignment correctness.
  3. Re-run execute flow with compliance-auditor oversight where required.
  4. Freeze production execute operations while governance prerequisites fail.

3.4 Recovery Verification Failures#

Trigger: repeated BACKUP_DR_VERIFICATION_FAILED outcomes.

Actions:

  1. Validate execution terminal state and verification payload (executionId, verificationId, unresolved issue counts).
  2. Inspect verification adapter health and post-recovery data consistency.
  3. Re-run verification after corrective actions on unresolved issues.
  4. Prevent incident closure until verification reaches deterministic pass state.

3.5 Disaster Recovery Incident Closure Failures#

Trigger: repeated BACKUP_DR_INCIDENT_RECOVERY_FAILED outcomes.

Actions:

  1. Validate incident lifecycle state, severity policy, and remediation action payload.
  2. Confirm recovery action and runbook steps are linked to traceable evidence.
  3. Re-run incident recovery under incident-command coordination.
  4. Keep release gate active until incident closure preconditions pass.

3.6 Telemetry and Audit Degradation#

Trigger: missing or delayed studio_backup_disaster_recovery_ux_* telemetry events.

Actions:

  1. Validate analytics sink health, client-side queue depth, and delivery backlog.
  2. Enable degraded-observability notice in Backup/DR workspace.
  3. Replay buffered telemetry where replay semantics are supported.
  4. Verify SLO panel continuity and audit log completeness after restoration.

4. Release Checklist#

  • Validate /studio/backup-disaster-recovery-ux renders and executes required flows.
  • Validate optimistic, error, and retry behavior across all Backup/DR flows.
  • Validate permission-tier, waiver, and production CAB policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility/performance evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Backup/DR Incident
Detected: {timestamp}
Impact: {backup-baseline | restore-simulation | execute-failover-restore | recovery-verification | incident-recovery}
Suspected cause: {restore-point integrity drift | policy-gate regression | execution failure | verification adapter degradation | telemetry degradation}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Backup/DR Update
Time: {timestamp}
Mitigation: {rollback | policy patch | simulation fallback | execution hold | release hold}
Affected flows: {defineBackupPlanBaseline | runRestoreSimulation | executeFailoverOrRestore | verifyRecoveryIntegrity | recoverDisasterRecoveryIncident}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Backup/DR Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Backup/DR workspace UX #oshun-web Platform on-call
Backup/restore orchestration APIs #oshun-bff BFF lead
Simulation/execute/recovery runtime #oshun-platform-sre SRE lead
Policy-tier/CAB/waiver governance control #oshun-security Security + Compliance
Telemetry and audit reporting continuity #oshun-data-platform Data platform lead

7. Acceptance Criteria (OST-00312)#

OST-00312 is complete only when:

  1. Ownership and decision authority are documented.
  2. Backup/DR SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.