Eve library

OSHUN Studio Multi-Project Operations Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0024-oshun-studio-multi-project-operations.md
  • docs/releases/v1/specs/studio-multi-project-operations-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-multi-project-operations-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioMultiProjectOperationsWorkspace.tsx
  • apps/oshun/web/src/analytics/studioMultiProjectOperationsTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio multi-project operations governance workflows.

This document is the execution artifact for OST-00096.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Multi-project operations route OSHUN Web Platform Web route + analytics contracts
Operation envelope and dependency model Studio Architecture + Program Ops Domain Integrators operation schema + dependency graph
Stage transition orchestration OSHUN BFF SRE readiness checks + transition service
Policy and approval controls Security + Program Leadership Ops entitlement + quorum policies
Telemetry and audit evidence Data Platform SRE event sinks + ingestion pipelines

2. Core SLOs#

Metric Target Alert Threshold
Multi-project flow success rate >= 99.0% < 98.0% over 15 min
Readiness gate integrity 100% deterministic any false pass
Stage transition consistency 100% valid transitions any invalid transition accepted
Multi-project flow p95 latency <= 2200ms > 3200ms over 15 min
Multi-project audit completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Baseline Publish Failures#

Trigger: repeated MULTI_PROJECT_BASELINE_FAILED outcomes.

Actions:

  1. Inspect operation envelope and dependency-graph diagnostics.
  2. Validate project-set uniqueness and release-window metadata.
  3. Re-run baseline publish in preview with validation logging.
  4. Roll back to last certified operation baseline if needed.

3.2 Readiness Stage Start Failures#

Trigger: repeated MULTI_PROJECT_READINESS_STAGE_FAILED outcomes.

Actions:

  1. Identify failing project readiness gates and dependency blockers.
  2. Remediate failed gates and stale readiness snapshots.
  3. Re-evaluate gate status.
  4. Re-attempt stage start and block promotion until pass.

3.3 Rollback Orchestration Failures#

Trigger: repeated MULTI_PROJECT_ROLLBACK_FAILED outcomes.

Actions:

  1. Validate rollback checkpoint integrity and dependency order.
  2. Recompute rollback sequence using dependency graph constraints.
  3. Execute rollback in controlled mode.
  4. Confirm deterministic recovery by project and stage.

3.4 Policy Denial Spike#

Trigger: unexpected increase in ROLE_POLICY_DENIED outcomes.

Actions:

  1. Compare flow policy map against entitlement contracts.
  2. Verify recent identity payload/version changes.
  3. Apply policy hotfix or rollback entitlement drift.
  4. Confirm deny-rate normalization after mitigation.

3.5 Telemetry and Audit Degradation#

Trigger: missing or delayed multi-project operation telemetry events.

Actions:

  1. Validate analytics sink health and queue depth.
  2. Enable degraded-observability warning in workspace.
  3. Reconcile missed events from buffered traces.
  4. Confirm event completeness and SLO recovery.

4. Release Checklist#

  • Validate /studio/multi-project-operations renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate role-policy allow/deny behavior.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Multi-Project Operations Incident
Detected: {timestamp}
Impact: {baseline/readiness/rollback impact}
Suspected cause: {dependency drift | gate failure | rollback failure | policy drift}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Multi-Project Operations Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {operationBaselinePublish | readinessGateStageStart | rollbackOrchestrationRecovery}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Multi-Project Operations Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and workspace #oshun-web Platform on-call
Program operations and dependencies #project-obsidian-ops Program director + architecture council
Stage orchestration and runtime #oshun-bff BFF architecture lead
Policy and permissions #oshun-security Security lead
Telemetry pipeline and reliability #oshun-data-platform SRE lead

7. Acceptance Criteria (OST-00096)#

OST-00096 is complete only when:

  1. Ownership and decision authority are documented.
  2. Multi-project operations SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.