# OSHUN Studio Multi-Project Operations Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0024-oshun-studio-multi-project-operations.md`
- `docs/releases/v1/specs/studio-multi-project-operations-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-multi-project-operations-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioMultiProjectOperationsWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioMultiProjectOperationsTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio multi-project operations governance workflows.

This document is the execution artifact for `OST-00096`.

## 1. Ownership Model

| Surface                                 | Primary Owner                     | Backup Owner       | Dependencies                          |
| --------------------------------------- | --------------------------------- | ------------------ | ------------------------------------- |
| Multi-project operations route          | OSHUN Web                         | Platform Web       | route + analytics contracts           |
| Operation envelope and dependency model | Studio Architecture + Program Ops | Domain Integrators | operation schema + dependency graph   |
| Stage transition orchestration          | OSHUN BFF                         | SRE                | readiness checks + transition service |
| Policy and approval controls            | Security + Program Leadership     | Ops                | entitlement + quorum policies         |
| Telemetry and audit evidence            | Data Platform                     | SRE                | event sinks + ingestion pipelines     |

## 2. Core SLOs

| Metric                           | Target                 | Alert Threshold                 |
| -------------------------------- | ---------------------- | ------------------------------- |
| Multi-project flow success rate  | >= 99.0%               | < 98.0% over 15 min             |
| Readiness gate integrity         | 100% deterministic     | any false pass                  |
| Stage transition consistency     | 100% valid transitions | any invalid transition accepted |
| Multi-project flow p95 latency   | <= 2200ms              | > 3200ms over 15 min            |
| Multi-project audit completeness | >= 99.5%               | < 99.0% over 15 min             |

## 3. Operational Playbooks

### 3.1 Baseline Publish Failures

Trigger: repeated `MULTI_PROJECT_BASELINE_FAILED` outcomes.

Actions:

1. Inspect operation envelope and dependency-graph diagnostics.
2. Validate project-set uniqueness and release-window metadata.
3. Re-run baseline publish in preview with validation logging.
4. Roll back to last certified operation baseline if needed.

### 3.2 Readiness Stage Start Failures

Trigger: repeated `MULTI_PROJECT_READINESS_STAGE_FAILED` outcomes.

Actions:

1. Identify failing project readiness gates and dependency blockers.
2. Remediate failed gates and stale readiness snapshots.
3. Re-evaluate gate status.
4. Re-attempt stage start and block promotion until pass.

### 3.3 Rollback Orchestration Failures

Trigger: repeated `MULTI_PROJECT_ROLLBACK_FAILED` outcomes.

Actions:

1. Validate rollback checkpoint integrity and dependency order.
2. Recompute rollback sequence using dependency graph constraints.
3. Execute rollback in controlled mode.
4. Confirm deterministic recovery by project and stage.

### 3.4 Policy Denial Spike

Trigger: unexpected increase in `ROLE_POLICY_DENIED` outcomes.

Actions:

1. Compare flow policy map against entitlement contracts.
2. Verify recent identity payload/version changes.
3. Apply policy hotfix or rollback entitlement drift.
4. Confirm deny-rate normalization after mitigation.

### 3.5 Telemetry and Audit Degradation

Trigger: missing or delayed multi-project operation telemetry events.

Actions:

1. Validate analytics sink health and queue depth.
2. Enable degraded-observability warning in workspace.
3. Reconcile missed events from buffered traces.
4. Confirm event completeness and SLO recovery.

## 4. Release Checklist

- [ ] Validate `/studio/multi-project-operations` renders and executes required
      flows.
- [ ] Validate optimistic, error, and retry behavior for each flow.
- [ ] Validate role-policy allow/deny behavior.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```
[P{severity}] Studio Multi-Project Operations Incident
Detected: {timestamp}
Impact: {baseline/readiness/rollback impact}
Suspected cause: {dependency drift | gate failure | rollback failure | policy drift}
Owner: {on-call}
```

### Incident Update

```
[P{severity}] Studio Multi-Project Operations Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {operationBaselinePublish | readinessGateStageStart | rollbackOrchestrationRecovery}
ETA: {time}
```

### Incident Resolved

```
[P{severity}] Studio Multi-Project Operations Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                                | Primary Channel         | Escalate To                             |
| ----------------------------------- | ----------------------- | --------------------------------------- |
| Web runtime and workspace           | `#oshun-web`            | Platform on-call                        |
| Program operations and dependencies | `#project-obsidian-ops` | Program director + architecture council |
| Stage orchestration and runtime     | `#oshun-bff`            | BFF architecture lead                   |
| Policy and permissions              | `#oshun-security`       | Security lead                           |
| Telemetry pipeline and reliability  | `#oshun-data-platform`  | SRE lead                                |

## 7. Acceptance Criteria (OST-00096)

`OST-00096` is complete only when:

1. Ownership and decision authority are documented.
2. Multi-project operations SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
