Eve library

OSHUN Studio Real-Time Collaboration Substrate Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0032-oshun-studio-real-time-collaboration-substrate.md
  • docs/releases/v1/specs/studio-real-time-collaboration-substrate-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-real-time-collaboration-substrate-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioRealTimeCollaborationSubstrateWorkspace.tsx
  • apps/oshun/web/src/analytics/studioRealTimeCollaborationSubstrateTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio real-time collaboration-substrate workflows.

This document is the execution artifact for OST-00160.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Collaboration workspace route OSHUN Web Platform Web route contracts + UI flow orchestration
Session topology and channel fan-out OSHUN BFF Data Platform websocket/session infrastructure
Conflict-resolution contract governance Domain Integrators Studio Architecture merge strategy adapters + policy metadata
Replay and recovery orchestration Reliability + BFF Platform SRE cursor persistence + replay services
Collaboration audit/governance exports Security + Compliance Data Platform analytics ingestion + retention controls

2. Core SLOs#

Metric Target Alert Threshold
Collaboration flow success rate >= 99.0% < 98.0% over 15 min
Session join/handshake success >= 99.5% < 99.0% over 15 min
Collaboration flow p95 latency <= 2400ms > 3400ms over 15 min
Conflict-resolution determinism 100% deterministic any non-deterministic resolution
Replay recovery completeness >= 99.5% < 99.0% over 15 min
Collaboration audit event completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Session Join and Capability Handshake Failures#

Trigger: repeated COLLAB_SESSION_JOIN_FAILED outcomes.

Actions:

  1. Validate workspace/session/channel identifiers and entitlement contract.
  2. Inspect capability grant policy for recent drift.
  3. Re-run handshake in preview with trace logging enabled.
  4. Block production rollout until handshake reliability returns to SLO.

3.2 Live Editing Flow Failures#

Trigger: repeated COLLAB_LIVE_EDITING_FAILED outcomes.

Actions:

  1. Validate event ordering contracts and sequence integrity.
  2. Inspect channel fan-out latency and dropped-message indicators.
  3. Re-run live edit flow with deterministic sequencing diagnostics.
  4. Degrade to read-only mode if state divergence risk increases.

3.3 Conflict Resolution Failures#

Trigger: repeated COLLAB_CONFLICT_RESOLUTION_FAILED outcomes.

Actions:

  1. Validate configured conflict strategy for affected entity type.
  2. Inspect merge outputs for non-deterministic behavior.
  3. Re-run conflict flow with manual strategy fallback.
  4. Escalate to architecture/security when deterministic guarantees fail.

3.4 Replay Recovery Failures#

Trigger: repeated COLLAB_REPLAY_RECOVERY_FAILED outcomes.

Actions:

  1. Validate replay cursor integrity and session continuity metadata.
  2. Inspect replay pipeline for ordering or pagination drift.
  3. Re-run recovery path with full snapshot fallback.
  4. Trigger incident if reconnect state divergence persists.

3.5 Governance Audit Export Failures#

Trigger: repeated COLLAB_GOVERNANCE_EXPORT_FAILED outcomes.

Actions:

  1. Validate export package ID and governance metadata completeness.
  2. Inspect retention tag and compliance schema validation output.
  3. Re-run export with security reviewer supervision.
  4. Block production exports until governance package checks pass.

3.6 Policy Denial Spike#

Trigger: unusual increase in ROLE_POLICY_DENIED, MANUAL_CONFLICT_POLICY_DENIED, or PRODUCTION_EXPORT_POLICY_DENIED.

Actions:

  1. Compare allowlists with current RBAC/policy bundles.
  2. Inspect recent entitlement changes for drift.
  3. Apply policy hotfix or roll back drifted policy release.
  4. Confirm deny-rate returns to baseline after mitigation.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed collaboration telemetry/audit events.

Actions:

  1. Validate analytics sink health and ingestion queue depth.
  2. Enable degraded-observability warning in collaboration workspace.
  3. Replay buffered telemetry events when supported.
  4. Confirm audit completeness and SLO data freshness recovery.

4. Release Checklist#

  • Validate /studio/real-time-collaboration-substrate renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate role-policy allow/deny behavior and manual-conflict policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Real-Time Collaboration Substrate Incident
Detected: {timestamp}
Impact: {session-join | live-editing | conflict-resolution | replay-recovery | governance-export}
Suspected cause: {policy drift | sequence failure | replay contract regression}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Real-Time Collaboration Substrate Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {sessionJoinCapabilityHandshake | liveMultiActorEditing | conflictResolutionOrchestration | disconnectReplayRecovery | governanceAuditEvidenceExport}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Real-Time Collaboration Substrate Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and collaboration workspace #oshun-web Platform on-call
Session topology and fan-out infrastructure #oshun-bff BFF lead
Conflict strategy and policy governance #oshun-security Security lead
Replay/recovery reliability #oshun-platform-sre SRE lead
Audit/compliance evidence pipelines #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00160)#

OST-00160 is complete only when:

  1. Ownership and decision authority are documented.
  2. Collaboration-substrate SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.