# OSHUN Studio Observability and Operational Dashboards Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN
Data/Analytics + OSHUN Security + Program Operations  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0041-oshun-studio-observability-and-operational-dashboards.md`
- `docs/releases/v1/specs/studio-observability-and-operational-dashboards-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-observability-and-operational-dashboards-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioObservabilityOperationalDashboardsWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioObservabilityOperationalDashboardsTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio observability and operational dashboard workflows.

This document is the execution artifact for `OST-00232`.

## 1. Ownership Model

| Surface                                 | Primary Owner            | Backup Owner          | Dependencies                                           |
| --------------------------------------- | ------------------------ | --------------------- | ------------------------------------------------------ |
| Observability dashboard workspace route | OSHUN Web                | Platform Web          | route contracts + dashboard UX orchestration           |
| Signal-card and SLO ingestion           | Data Platform + BFF      | Platform SRE          | metrics streams + schema/contract registries           |
| Dependency drill-down services          | Platform SRE + BFF       | Domain service owners | topology graph + trace/context adapters                |
| Remediation and lifecycle policy guard  | Security + Compliance    | Program Operations    | RBAC/policy tier bundles + CAB governance contracts    |
| Readiness-gate decisioning              | Program Ops + Compliance | Platform SRE          | SLO rollups + telemetry quality + release orchestrator |
| Audit and evidence export stream        | Data Platform + Security | Platform SRE          | analytics ingestion + retention/export controls        |

## 2. Core SLOs

| Metric                                     | Target    | Alert Threshold      |
| ------------------------------------------ | --------- | -------------------- |
| Dashboard flow success rate                | >= 99.0%  | < 98.0% over 15 min  |
| Signal-card freshness reliability          | >= 99.5%  | < 99.0% over 15 min  |
| Dashboard flow p95 latency                 | <= 2500ms | > 3500ms over 15 min |
| Alert lifecycle transition integrity       | >= 99.5%  | < 99.0% over 15 min  |
| Remediation policy enforcement correctness | >= 99.5%  | < 99.0% over 15 min  |
| Readiness-gate decision determinism        | >= 99.5%  | < 99.0% over 15 min  |
| Telemetry/audit event continuity           | >= 99.5%  | < 99.0% over 15 min  |

## 3. Operational Playbooks

### 3.1 Signal Ingestion Failures

Trigger: repeated `DASHBOARD_SIGNAL_INGEST_FAILED` outcomes.

Actions:

1. Validate signal payload identity and required fields (`signalId`,
   `signalType`, `status`, `window`).
2. Inspect SLI/SLO source stream schema compatibility and freshness.
3. Re-run ingestion in preview channel with contract diagnostics.
4. Block production readiness decisions until ingestion stability recovers.

### 3.2 Dependency Drilldown Failures

Trigger: repeated `DASHBOARD_DEPENDENCY_DRILLDOWN_FAILED` outcomes.

Actions:

1. Validate dependency graph and trace-reference payload integrity.
2. Inspect graph service latency/error rates and stale topology snapshots.
3. Re-run drilldown with fallback graph query and bounded window scope.
4. Escalate to Platform SRE if root-cause isolation remains unavailable.

### 3.3 Remediation Execution Failures

Trigger: repeated `DASHBOARD_REMEDIATION_FAILED` outcomes.

Actions:

1. Validate remediation action payload and alert linkage.
2. Inspect policy decision logs for role/tier/CAB denial mismatches.
3. Re-run remediation with approved fallback action type.
4. Freeze production-impacting remediation actions when policy integrity
   degrades.

### 3.4 Alert Lifecycle Transition Failures

Trigger: repeated `DASHBOARD_ALERT_LIFECYCLE_TRANSITION_FAILED` outcomes.

Actions:

1. Validate source/target lifecycle state semantics.
2. Inspect concurrency conflicts and stale-owner updates.
3. Re-run transition with optimistic concurrency metadata.
4. Escalate to incident commander if state machine becomes inconsistent.

### 3.5 Readiness-Gate Evaluation Failures

Trigger: repeated `DASHBOARD_READINESS_GATE_EVALUATION_FAILED` outcomes.

Actions:

1. Validate critical-alert and telemetry-quality preconditions.
2. Inspect SLO rollup freshness and completeness signals.
3. Re-run readiness gate with compliance auditor supervision.
4. Enforce release hold until deterministic gate verdict is restored.

### 3.6 Policy Denial Spike

Trigger: sustained increase in `ROLE_POLICY_DENIED`,
`PERMISSION_TIER_ROLE_DENIED`, `FLOW_PERMISSION_TIER_INSUFFICIENT`,
`POLICY_JUSTIFICATION_REQUIRED`,
`HIGH_IMPACT_REMEDIATION_REQUIRES_RESTRICTED_TIER`,
`PRODUCTION_APPROVAL_TICKET_REQUIRED`, or
`PRODUCTION_READINESS_GATE_REQUIRES_COMPLIANCE_AUDITOR`.

Actions:

1. Compare active policy bundles against expected role/tier matrix.
2. Inspect recent policy release changes for drift.
3. Roll back or patch policy bundle and verify deny-path normalization.
4. Revalidate remediation and readiness-gate flows post-fix.

### 3.7 Telemetry and Audit Degradation

Trigger: missing or delayed `studio_observability_operational_dashboards_*`
telemetry events.

Actions:

1. Validate analytics sink health and backlog depth.
2. Enable degraded-observability warning in dashboard workspace.
3. Replay buffered events where replay semantics are supported.
4. Confirm SLO panel and audit evidence completeness after recovery.

## 4. Release Checklist

- [ ] Validate `/studio/observability-operational-dashboards` renders and
      executes required flows.
- [ ] Validate optimistic, error, and retry behavior for all dashboard flows.
- [ ] Validate permission-tier, high-impact remediation, and production CAB
      policy enforcement.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility/performance evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```
[P{severity}] Studio Observability Dashboard Incident
Detected: {timestamp}
Impact: {signal-ingest | dependency-drilldown | remediation | alert-lifecycle | readiness-gate}
Suspected cause: {policy drift | schema regression | graph degradation | telemetry gap}
Owner: {on-call}
```

### Incident Update

```
[P{severity}] Studio Observability Dashboard Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {ingestDashboardSignalCard | runDependencyRootCauseDrilldown | executePolicyAwareRemediation | transitionAlertLifecycle | evaluateOperationalReadinessGate}
ETA: {time}
```

### Incident Resolved

```
[P{severity}] Studio Observability Dashboard Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                                  | Primary Channel        | Escalate To            |
| ------------------------------------- | ---------------------- | ---------------------- |
| Dashboard UX/runtime                  | `#oshun-web`           | Platform on-call       |
| Signal and SLO ingestion              | `#oshun-data-platform` | Data platform lead     |
| Dependency graph/drilldown services   | `#oshun-platform-sre`  | SRE lead               |
| Policy/CAB governance controls        | `#oshun-security`      | Security + Compliance  |
| Alert triage and readiness operations | `#oshun-program-ops`   | Program Ops leadership |

## 7. Acceptance Criteria (OST-00232)

`OST-00232` is complete only when:

1. Ownership and decision authority are documented.
2. Observability/dashboard SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
