Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security +
Program Operations
Last Updated: 2026-02-23
Related:
docs/adr/ADR-0032-oshun-studio-real-time-collaboration-substrate.mddocs/releases/v1/specs/studio-real-time-collaboration-substrate-dependency-and-data-contract-map.mddocs/releases/v1/specs/studio-real-time-collaboration-substrate-user-journeys-and-edge-cases.mdapps/oshun/web/src/components/studio/StudioRealTimeCollaborationSubstrateWorkspace.tsxapps/oshun/web/src/analytics/studioRealTimeCollaborationSubstrateTelemetry.ts
Purpose#
Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio real-time collaboration-substrate workflows.
This document is the execution artifact for OST-00160.
1. Ownership Model#
| Surface | Primary Owner | Backup Owner | Dependencies |
|---|---|---|---|
| Collaboration workspace route | OSHUN Web | Platform Web | route contracts + UI flow orchestration |
| Session topology and channel fan-out | OSHUN BFF | Data Platform | websocket/session infrastructure |
| Conflict-resolution contract governance | Domain Integrators | Studio Architecture | merge strategy adapters + policy metadata |
| Replay and recovery orchestration | Reliability + BFF | Platform SRE | cursor persistence + replay services |
| Collaboration audit/governance exports | Security + Compliance | Data Platform | analytics ingestion + retention controls |
2. Core SLOs#
| Metric | Target | Alert Threshold |
|---|---|---|
| Collaboration flow success rate | >= 99.0% | < 98.0% over 15 min |
| Session join/handshake success | >= 99.5% | < 99.0% over 15 min |
| Collaboration flow p95 latency | <= 2400ms | > 3400ms over 15 min |
| Conflict-resolution determinism | 100% deterministic | any non-deterministic resolution |
| Replay recovery completeness | >= 99.5% | < 99.0% over 15 min |
| Collaboration audit event completeness | >= 99.5% | < 99.0% over 15 min |
3. Operational Playbooks#
3.1 Session Join and Capability Handshake Failures#
Trigger: repeated COLLAB_SESSION_JOIN_FAILED outcomes.
Actions:
- Validate workspace/session/channel identifiers and entitlement contract.
- Inspect capability grant policy for recent drift.
- Re-run handshake in preview with trace logging enabled.
- Block production rollout until handshake reliability returns to SLO.
3.2 Live Editing Flow Failures#
Trigger: repeated COLLAB_LIVE_EDITING_FAILED outcomes.
Actions:
- Validate event ordering contracts and sequence integrity.
- Inspect channel fan-out latency and dropped-message indicators.
- Re-run live edit flow with deterministic sequencing diagnostics.
- Degrade to read-only mode if state divergence risk increases.
3.3 Conflict Resolution Failures#
Trigger: repeated COLLAB_CONFLICT_RESOLUTION_FAILED outcomes.
Actions:
- Validate configured conflict strategy for affected entity type.
- Inspect merge outputs for non-deterministic behavior.
- Re-run conflict flow with manual strategy fallback.
- Escalate to architecture/security when deterministic guarantees fail.
3.4 Replay Recovery Failures#
Trigger: repeated COLLAB_REPLAY_RECOVERY_FAILED outcomes.
Actions:
- Validate replay cursor integrity and session continuity metadata.
- Inspect replay pipeline for ordering or pagination drift.
- Re-run recovery path with full snapshot fallback.
- Trigger incident if reconnect state divergence persists.
3.5 Governance Audit Export Failures#
Trigger: repeated COLLAB_GOVERNANCE_EXPORT_FAILED outcomes.
Actions:
- Validate export package ID and governance metadata completeness.
- Inspect retention tag and compliance schema validation output.
- Re-run export with security reviewer supervision.
- Block production exports until governance package checks pass.
3.6 Policy Denial Spike#
Trigger: unusual increase in ROLE_POLICY_DENIED,
MANUAL_CONFLICT_POLICY_DENIED, or PRODUCTION_EXPORT_POLICY_DENIED.
Actions:
- Compare allowlists with current RBAC/policy bundles.
- Inspect recent entitlement changes for drift.
- Apply policy hotfix or roll back drifted policy release.
- Confirm deny-rate returns to baseline after mitigation.
3.7 Telemetry and Audit Degradation#
Trigger: missing or delayed collaboration telemetry/audit events.
Actions:
- Validate analytics sink health and ingestion queue depth.
- Enable degraded-observability warning in collaboration workspace.
- Replay buffered telemetry events when supported.
- Confirm audit completeness and SLO data freshness recovery.
4. Release Checklist#
- Validate
/studio/real-time-collaboration-substraterenders and executes required flows. - Validate optimistic, error, and retry behavior for each flow.
- Validate role-policy allow/deny behavior and manual-conflict policy enforcement.
- Validate telemetry/audit emissions and SLO panel updates.
- Validate integration/e2e/accessibility evidence artifacts.
5. Incident Communication Templates#
Incident Start#
[P{severity}] Studio Real-Time Collaboration Substrate Incident
Detected: {timestamp}
Impact: {session-join | live-editing | conflict-resolution | replay-recovery | governance-export}
Suspected cause: {policy drift | sequence failure | replay contract regression}
Owner: {on-call}
Incident Update#
[P{severity}] Studio Real-Time Collaboration Substrate Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {sessionJoinCapabilityHandshake | liveMultiActorEditing | conflictResolutionOrchestration | disconnectReplayRecovery | governanceAuditEvidenceExport}
ETA: {time}
Incident Resolved#
[P{severity}] Studio Real-Time Collaboration Substrate Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
6. Escalation Paths#
| Area | Primary Channel | Escalate To |
|---|---|---|
| Web runtime and collaboration workspace | #oshun-web |
Platform on-call |
| Session topology and fan-out infrastructure | #oshun-bff |
BFF lead |
| Conflict strategy and policy governance | #oshun-security |
Security lead |
| Replay/recovery reliability | #oshun-platform-sre |
SRE lead |
| Audit/compliance evidence pipelines | #oshun-data-platform |
Data platform/SRE lead |
7. Acceptance Criteria (OST-00160)#
OST-00160 is complete only when:
- Ownership and decision authority are documented.
- Collaboration-substrate SLOs and alert thresholds are documented.
- At least five playbooks are documented for major failure modes.
- Release checklist, communication templates, and escalation paths exist.