# OSHUN Studio Real-Time Collaboration Substrate Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security +
Program Operations  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0032-oshun-studio-real-time-collaboration-substrate.md`
- `docs/releases/v1/specs/studio-real-time-collaboration-substrate-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-real-time-collaboration-substrate-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioRealTimeCollaborationSubstrateWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioRealTimeCollaborationSubstrateTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio real-time collaboration-substrate workflows.

This document is the execution artifact for `OST-00160`.

## 1. Ownership Model

| Surface                                 | Primary Owner         | Backup Owner        | Dependencies                              |
| --------------------------------------- | --------------------- | ------------------- | ----------------------------------------- |
| Collaboration workspace route           | OSHUN Web             | Platform Web        | route contracts + UI flow orchestration   |
| Session topology and channel fan-out    | OSHUN BFF             | Data Platform       | websocket/session infrastructure          |
| Conflict-resolution contract governance | Domain Integrators    | Studio Architecture | merge strategy adapters + policy metadata |
| Replay and recovery orchestration       | Reliability + BFF     | Platform SRE        | cursor persistence + replay services      |
| Collaboration audit/governance exports  | Security + Compliance | Data Platform       | analytics ingestion + retention controls  |

## 2. Core SLOs

| Metric                                 | Target             | Alert Threshold                  |
| -------------------------------------- | ------------------ | -------------------------------- |
| Collaboration flow success rate        | >= 99.0%           | < 98.0% over 15 min              |
| Session join/handshake success         | >= 99.5%           | < 99.0% over 15 min              |
| Collaboration flow p95 latency         | <= 2400ms          | > 3400ms over 15 min             |
| Conflict-resolution determinism        | 100% deterministic | any non-deterministic resolution |
| Replay recovery completeness           | >= 99.5%           | < 99.0% over 15 min              |
| Collaboration audit event completeness | >= 99.5%           | < 99.0% over 15 min              |

## 3. Operational Playbooks

### 3.1 Session Join and Capability Handshake Failures

Trigger: repeated `COLLAB_SESSION_JOIN_FAILED` outcomes.

Actions:

1. Validate workspace/session/channel identifiers and entitlement contract.
2. Inspect capability grant policy for recent drift.
3. Re-run handshake in preview with trace logging enabled.
4. Block production rollout until handshake reliability returns to SLO.

### 3.2 Live Editing Flow Failures

Trigger: repeated `COLLAB_LIVE_EDITING_FAILED` outcomes.

Actions:

1. Validate event ordering contracts and sequence integrity.
2. Inspect channel fan-out latency and dropped-message indicators.
3. Re-run live edit flow with deterministic sequencing diagnostics.
4. Degrade to read-only mode if state divergence risk increases.

### 3.3 Conflict Resolution Failures

Trigger: repeated `COLLAB_CONFLICT_RESOLUTION_FAILED` outcomes.

Actions:

1. Validate configured conflict strategy for affected entity type.
2. Inspect merge outputs for non-deterministic behavior.
3. Re-run conflict flow with manual strategy fallback.
4. Escalate to architecture/security when deterministic guarantees fail.

### 3.4 Replay Recovery Failures

Trigger: repeated `COLLAB_REPLAY_RECOVERY_FAILED` outcomes.

Actions:

1. Validate replay cursor integrity and session continuity metadata.
2. Inspect replay pipeline for ordering or pagination drift.
3. Re-run recovery path with full snapshot fallback.
4. Trigger incident if reconnect state divergence persists.

### 3.5 Governance Audit Export Failures

Trigger: repeated `COLLAB_GOVERNANCE_EXPORT_FAILED` outcomes.

Actions:

1. Validate export package ID and governance metadata completeness.
2. Inspect retention tag and compliance schema validation output.
3. Re-run export with security reviewer supervision.
4. Block production exports until governance package checks pass.

### 3.6 Policy Denial Spike

Trigger: unusual increase in `ROLE_POLICY_DENIED`,
`MANUAL_CONFLICT_POLICY_DENIED`, or `PRODUCTION_EXPORT_POLICY_DENIED`.

Actions:

1. Compare allowlists with current RBAC/policy bundles.
2. Inspect recent entitlement changes for drift.
3. Apply policy hotfix or roll back drifted policy release.
4. Confirm deny-rate returns to baseline after mitigation.

### 3.7 Telemetry and Audit Degradation

Trigger: missing or delayed collaboration telemetry/audit events.

Actions:

1. Validate analytics sink health and ingestion queue depth.
2. Enable degraded-observability warning in collaboration workspace.
3. Replay buffered telemetry events when supported.
4. Confirm audit completeness and SLO data freshness recovery.

## 4. Release Checklist

- [ ] Validate `/studio/real-time-collaboration-substrate` renders and executes
      required flows.
- [ ] Validate optimistic, error, and retry behavior for each flow.
- [ ] Validate role-policy allow/deny behavior and manual-conflict policy
      enforcement.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```
[P{severity}] Studio Real-Time Collaboration Substrate Incident
Detected: {timestamp}
Impact: {session-join | live-editing | conflict-resolution | replay-recovery | governance-export}
Suspected cause: {policy drift | sequence failure | replay contract regression}
Owner: {on-call}
```

### Incident Update

```
[P{severity}] Studio Real-Time Collaboration Substrate Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {sessionJoinCapabilityHandshake | liveMultiActorEditing | conflictResolutionOrchestration | disconnectReplayRecovery | governanceAuditEvidenceExport}
ETA: {time}
```

### Incident Resolved

```
[P{severity}] Studio Real-Time Collaboration Substrate Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                                        | Primary Channel        | Escalate To            |
| ------------------------------------------- | ---------------------- | ---------------------- |
| Web runtime and collaboration workspace     | `#oshun-web`           | Platform on-call       |
| Session topology and fan-out infrastructure | `#oshun-bff`           | BFF lead               |
| Conflict strategy and policy governance     | `#oshun-security`      | Security lead          |
| Replay/recovery reliability                 | `#oshun-platform-sre`  | SRE lead               |
| Audit/compliance evidence pipelines         | `#oshun-data-platform` | Data platform/SRE lead |

## 7. Acceptance Criteria (OST-00160)

`OST-00160` is complete only when:

1. Ownership and decision authority are documented.
2. Collaboration-substrate SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
