Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN
Data/Analytics + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:
docs/adr/ADR-0041-oshun-studio-observability-and-operational-dashboards.mddocs/releases/v1/specs/studio-observability-and-operational-dashboards-dependency-and-data-contract-map.mddocs/releases/v1/specs/studio-observability-and-operational-dashboards-user-journeys-and-edge-cases.mdapps/oshun/web/src/components/studio/StudioObservabilityOperationalDashboardsWorkspace.tsxapps/oshun/web/src/analytics/studioObservabilityOperationalDashboardsTelemetry.ts
Purpose#
Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio observability and operational dashboard workflows.
This document is the execution artifact for OST-00232.
1. Ownership Model#
| Surface | Primary Owner | Backup Owner | Dependencies |
|---|---|---|---|
| Observability dashboard workspace route | OSHUN Web | Platform Web | route contracts + dashboard UX orchestration |
| Signal-card and SLO ingestion | Data Platform + BFF | Platform SRE | metrics streams + schema/contract registries |
| Dependency drill-down services | Platform SRE + BFF | Domain service owners | topology graph + trace/context adapters |
| Remediation and lifecycle policy guard | Security + Compliance | Program Operations | RBAC/policy tier bundles + CAB governance contracts |
| Readiness-gate decisioning | Program Ops + Compliance | Platform SRE | SLO rollups + telemetry quality + release orchestrator |
| Audit and evidence export stream | Data Platform + Security | Platform SRE | analytics ingestion + retention/export controls |
2. Core SLOs#
| Metric | Target | Alert Threshold |
|---|---|---|
| Dashboard flow success rate | >= 99.0% | < 98.0% over 15 min |
| Signal-card freshness reliability | >= 99.5% | < 99.0% over 15 min |
| Dashboard flow p95 latency | <= 2500ms | > 3500ms over 15 min |
| Alert lifecycle transition integrity | >= 99.5% | < 99.0% over 15 min |
| Remediation policy enforcement correctness | >= 99.5% | < 99.0% over 15 min |
| Readiness-gate decision determinism | >= 99.5% | < 99.0% over 15 min |
| Telemetry/audit event continuity | >= 99.5% | < 99.0% over 15 min |
3. Operational Playbooks#
3.1 Signal Ingestion Failures#
Trigger: repeated DASHBOARD_SIGNAL_INGEST_FAILED outcomes.
Actions:
- Validate signal payload identity and required fields (
signalId,signalType,status,window). - Inspect SLI/SLO source stream schema compatibility and freshness.
- Re-run ingestion in preview channel with contract diagnostics.
- Block production readiness decisions until ingestion stability recovers.
3.2 Dependency Drilldown Failures#
Trigger: repeated DASHBOARD_DEPENDENCY_DRILLDOWN_FAILED outcomes.
Actions:
- Validate dependency graph and trace-reference payload integrity.
- Inspect graph service latency/error rates and stale topology snapshots.
- Re-run drilldown with fallback graph query and bounded window scope.
- Escalate to Platform SRE if root-cause isolation remains unavailable.
3.3 Remediation Execution Failures#
Trigger: repeated DASHBOARD_REMEDIATION_FAILED outcomes.
Actions:
- Validate remediation action payload and alert linkage.
- Inspect policy decision logs for role/tier/CAB denial mismatches.
- Re-run remediation with approved fallback action type.
- Freeze production-impacting remediation actions when policy integrity degrades.
3.4 Alert Lifecycle Transition Failures#
Trigger: repeated DASHBOARD_ALERT_LIFECYCLE_TRANSITION_FAILED outcomes.
Actions:
- Validate source/target lifecycle state semantics.
- Inspect concurrency conflicts and stale-owner updates.
- Re-run transition with optimistic concurrency metadata.
- Escalate to incident commander if state machine becomes inconsistent.
3.5 Readiness-Gate Evaluation Failures#
Trigger: repeated DASHBOARD_READINESS_GATE_EVALUATION_FAILED outcomes.
Actions:
- Validate critical-alert and telemetry-quality preconditions.
- Inspect SLO rollup freshness and completeness signals.
- Re-run readiness gate with compliance auditor supervision.
- Enforce release hold until deterministic gate verdict is restored.
3.6 Policy Denial Spike#
Trigger: sustained increase in ROLE_POLICY_DENIED,
PERMISSION_TIER_ROLE_DENIED, FLOW_PERMISSION_TIER_INSUFFICIENT,
POLICY_JUSTIFICATION_REQUIRED,
HIGH_IMPACT_REMEDIATION_REQUIRES_RESTRICTED_TIER,
PRODUCTION_APPROVAL_TICKET_REQUIRED, or
PRODUCTION_READINESS_GATE_REQUIRES_COMPLIANCE_AUDITOR.
Actions:
- Compare active policy bundles against expected role/tier matrix.
- Inspect recent policy release changes for drift.
- Roll back or patch policy bundle and verify deny-path normalization.
- Revalidate remediation and readiness-gate flows post-fix.
3.7 Telemetry and Audit Degradation#
Trigger: missing or delayed studio_observability_operational_dashboards_*
telemetry events.
Actions:
- Validate analytics sink health and backlog depth.
- Enable degraded-observability warning in dashboard workspace.
- Replay buffered events where replay semantics are supported.
- Confirm SLO panel and audit evidence completeness after recovery.
4. Release Checklist#
- Validate
/studio/observability-operational-dashboardsrenders and executes required flows. - Validate optimistic, error, and retry behavior for all dashboard flows.
- Validate permission-tier, high-impact remediation, and production CAB policy enforcement.
- Validate telemetry/audit emissions and SLO panel updates.
- Validate integration/e2e/accessibility/performance evidence artifacts.
5. Incident Communication Templates#
Incident Start#
[P{severity}] Studio Observability Dashboard Incident
Detected: {timestamp}
Impact: {signal-ingest | dependency-drilldown | remediation | alert-lifecycle | readiness-gate}
Suspected cause: {policy drift | schema regression | graph degradation | telemetry gap}
Owner: {on-call}
Incident Update#
[P{severity}] Studio Observability Dashboard Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {ingestDashboardSignalCard | runDependencyRootCauseDrilldown | executePolicyAwareRemediation | transitionAlertLifecycle | evaluateOperationalReadinessGate}
ETA: {time}
Incident Resolved#
[P{severity}] Studio Observability Dashboard Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
6. Escalation Paths#
| Area | Primary Channel | Escalate To |
|---|---|---|
| Dashboard UX/runtime | #oshun-web |
Platform on-call |
| Signal and SLO ingestion | #oshun-data-platform |
Data platform lead |
| Dependency graph/drilldown services | #oshun-platform-sre |
SRE lead |
| Policy/CAB governance controls | #oshun-security |
Security + Compliance |
| Alert triage and readiness operations | #oshun-program-ops |
Program Ops leadership |
7. Acceptance Criteria (OST-00232)#
OST-00232 is complete only when:
- Ownership and decision authority are documented.
- Observability/dashboard SLOs and alert thresholds are documented.
- At least five playbooks are documented for major failure modes.
- Release checklist, communication templates, and escalation paths exist.