Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF
Last Updated: 2026-02-23
Related:
docs/adr/ADR-0024-oshun-studio-multi-project-operations.mddocs/releases/v1/specs/studio-multi-project-operations-dependency-and-data-contract-map.mddocs/releases/v1/specs/studio-multi-project-operations-user-journeys-and-edge-cases.mdapps/oshun/web/src/components/studio/StudioMultiProjectOperationsWorkspace.tsxapps/oshun/web/src/analytics/studioMultiProjectOperationsTelemetry.ts
Purpose#
Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio multi-project operations governance workflows.
This document is the execution artifact for OST-00096.
1. Ownership Model#
| Surface | Primary Owner | Backup Owner | Dependencies |
|---|---|---|---|
| Multi-project operations route | OSHUN Web | Platform Web | route + analytics contracts |
| Operation envelope and dependency model | Studio Architecture + Program Ops | Domain Integrators | operation schema + dependency graph |
| Stage transition orchestration | OSHUN BFF | SRE | readiness checks + transition service |
| Policy and approval controls | Security + Program Leadership | Ops | entitlement + quorum policies |
| Telemetry and audit evidence | Data Platform | SRE | event sinks + ingestion pipelines |
2. Core SLOs#
| Metric | Target | Alert Threshold |
|---|---|---|
| Multi-project flow success rate | >= 99.0% | < 98.0% over 15 min |
| Readiness gate integrity | 100% deterministic | any false pass |
| Stage transition consistency | 100% valid transitions | any invalid transition accepted |
| Multi-project flow p95 latency | <= 2200ms | > 3200ms over 15 min |
| Multi-project audit completeness | >= 99.5% | < 99.0% over 15 min |
3. Operational Playbooks#
3.1 Baseline Publish Failures#
Trigger: repeated MULTI_PROJECT_BASELINE_FAILED outcomes.
Actions:
- Inspect operation envelope and dependency-graph diagnostics.
- Validate project-set uniqueness and release-window metadata.
- Re-run baseline publish in preview with validation logging.
- Roll back to last certified operation baseline if needed.
3.2 Readiness Stage Start Failures#
Trigger: repeated MULTI_PROJECT_READINESS_STAGE_FAILED outcomes.
Actions:
- Identify failing project readiness gates and dependency blockers.
- Remediate failed gates and stale readiness snapshots.
- Re-evaluate gate status.
- Re-attempt stage start and block promotion until pass.
3.3 Rollback Orchestration Failures#
Trigger: repeated MULTI_PROJECT_ROLLBACK_FAILED outcomes.
Actions:
- Validate rollback checkpoint integrity and dependency order.
- Recompute rollback sequence using dependency graph constraints.
- Execute rollback in controlled mode.
- Confirm deterministic recovery by project and stage.
3.4 Policy Denial Spike#
Trigger: unexpected increase in ROLE_POLICY_DENIED outcomes.
Actions:
- Compare flow policy map against entitlement contracts.
- Verify recent identity payload/version changes.
- Apply policy hotfix or rollback entitlement drift.
- Confirm deny-rate normalization after mitigation.
3.5 Telemetry and Audit Degradation#
Trigger: missing or delayed multi-project operation telemetry events.
Actions:
- Validate analytics sink health and queue depth.
- Enable degraded-observability warning in workspace.
- Reconcile missed events from buffered traces.
- Confirm event completeness and SLO recovery.
4. Release Checklist#
- Validate
/studio/multi-project-operationsrenders and executes required flows. - Validate optimistic, error, and retry behavior for each flow.
- Validate role-policy allow/deny behavior.
- Validate telemetry/audit emissions and SLO panel updates.
- Validate integration/e2e/accessibility evidence artifacts.
5. Incident Communication Templates#
Incident Start#
text
[P{severity}] Studio Multi-Project Operations Incident
Detected: {timestamp}
Impact: {baseline/readiness/rollback impact}
Suspected cause: {dependency drift | gate failure | rollback failure | policy drift}
Owner: {on-call}
Incident Update#
text
[P{severity}] Studio Multi-Project Operations Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {operationBaselinePublish | readinessGateStageStart | rollbackOrchestrationRecovery}
ETA: {time}
Incident Resolved#
text
[P{severity}] Studio Multi-Project Operations Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
6. Escalation Paths#
| Area | Primary Channel | Escalate To |
|---|---|---|
| Web runtime and workspace | #oshun-web |
Platform on-call |
| Program operations and dependencies | #project-obsidian-ops |
Program director + architecture council |
| Stage orchestration and runtime | #oshun-bff |
BFF architecture lead |
| Policy and permissions | #oshun-security |
Security lead |
| Telemetry pipeline and reliability | #oshun-data-platform |
SRE lead |
7. Acceptance Criteria (OST-00096)#
OST-00096 is complete only when:
- Ownership and decision authority are documented.
- Multi-project operations SLOs and alert thresholds are documented.
- At least five playbooks are documented for major failure modes.
- Release checklist, communication templates, and escalation paths exist.