Eve library

OSHUN Studio Observability and Operational Dashboards Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN Data/Analytics + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0041-oshun-studio-observability-and-operational-dashboards.md
  • docs/releases/v1/specs/studio-observability-and-operational-dashboards-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-observability-and-operational-dashboards-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioObservabilityOperationalDashboardsWorkspace.tsx
  • apps/oshun/web/src/analytics/studioObservabilityOperationalDashboardsTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio observability and operational dashboard workflows.

This document is the execution artifact for OST-00232.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Observability dashboard workspace route OSHUN Web Platform Web route contracts + dashboard UX orchestration
Signal-card and SLO ingestion Data Platform + BFF Platform SRE metrics streams + schema/contract registries
Dependency drill-down services Platform SRE + BFF Domain service owners topology graph + trace/context adapters
Remediation and lifecycle policy guard Security + Compliance Program Operations RBAC/policy tier bundles + CAB governance contracts
Readiness-gate decisioning Program Ops + Compliance Platform SRE SLO rollups + telemetry quality + release orchestrator
Audit and evidence export stream Data Platform + Security Platform SRE analytics ingestion + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
Dashboard flow success rate >= 99.0% < 98.0% over 15 min
Signal-card freshness reliability >= 99.5% < 99.0% over 15 min
Dashboard flow p95 latency <= 2500ms > 3500ms over 15 min
Alert lifecycle transition integrity >= 99.5% < 99.0% over 15 min
Remediation policy enforcement correctness >= 99.5% < 99.0% over 15 min
Readiness-gate decision determinism >= 99.5% < 99.0% over 15 min
Telemetry/audit event continuity >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Signal Ingestion Failures#

Trigger: repeated DASHBOARD_SIGNAL_INGEST_FAILED outcomes.

Actions:

  1. Validate signal payload identity and required fields (signalId, signalType, status, window).
  2. Inspect SLI/SLO source stream schema compatibility and freshness.
  3. Re-run ingestion in preview channel with contract diagnostics.
  4. Block production readiness decisions until ingestion stability recovers.

3.2 Dependency Drilldown Failures#

Trigger: repeated DASHBOARD_DEPENDENCY_DRILLDOWN_FAILED outcomes.

Actions:

  1. Validate dependency graph and trace-reference payload integrity.
  2. Inspect graph service latency/error rates and stale topology snapshots.
  3. Re-run drilldown with fallback graph query and bounded window scope.
  4. Escalate to Platform SRE if root-cause isolation remains unavailable.

3.3 Remediation Execution Failures#

Trigger: repeated DASHBOARD_REMEDIATION_FAILED outcomes.

Actions:

  1. Validate remediation action payload and alert linkage.
  2. Inspect policy decision logs for role/tier/CAB denial mismatches.
  3. Re-run remediation with approved fallback action type.
  4. Freeze production-impacting remediation actions when policy integrity degrades.

3.4 Alert Lifecycle Transition Failures#

Trigger: repeated DASHBOARD_ALERT_LIFECYCLE_TRANSITION_FAILED outcomes.

Actions:

  1. Validate source/target lifecycle state semantics.
  2. Inspect concurrency conflicts and stale-owner updates.
  3. Re-run transition with optimistic concurrency metadata.
  4. Escalate to incident commander if state machine becomes inconsistent.

3.5 Readiness-Gate Evaluation Failures#

Trigger: repeated DASHBOARD_READINESS_GATE_EVALUATION_FAILED outcomes.

Actions:

  1. Validate critical-alert and telemetry-quality preconditions.
  2. Inspect SLO rollup freshness and completeness signals.
  3. Re-run readiness gate with compliance auditor supervision.
  4. Enforce release hold until deterministic gate verdict is restored.

3.6 Policy Denial Spike#

Trigger: sustained increase in ROLE_POLICY_DENIED, PERMISSION_TIER_ROLE_DENIED, FLOW_PERMISSION_TIER_INSUFFICIENT, POLICY_JUSTIFICATION_REQUIRED, HIGH_IMPACT_REMEDIATION_REQUIRES_RESTRICTED_TIER, PRODUCTION_APPROVAL_TICKET_REQUIRED, or PRODUCTION_READINESS_GATE_REQUIRES_COMPLIANCE_AUDITOR.

Actions:

  1. Compare active policy bundles against expected role/tier matrix.
  2. Inspect recent policy release changes for drift.
  3. Roll back or patch policy bundle and verify deny-path normalization.
  4. Revalidate remediation and readiness-gate flows post-fix.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed studio_observability_operational_dashboards_* telemetry events.

Actions:

  1. Validate analytics sink health and backlog depth.
  2. Enable degraded-observability warning in dashboard workspace.
  3. Replay buffered events where replay semantics are supported.
  4. Confirm SLO panel and audit evidence completeness after recovery.

4. Release Checklist#

  • Validate /studio/observability-operational-dashboards renders and executes required flows.
  • Validate optimistic, error, and retry behavior for all dashboard flows.
  • Validate permission-tier, high-impact remediation, and production CAB policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility/performance evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Observability Dashboard Incident
Detected: {timestamp}
Impact: {signal-ingest | dependency-drilldown | remediation | alert-lifecycle | readiness-gate}
Suspected cause: {policy drift | schema regression | graph degradation | telemetry gap}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Observability Dashboard Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {ingestDashboardSignalCard | runDependencyRootCauseDrilldown | executePolicyAwareRemediation | transitionAlertLifecycle | evaluateOperationalReadinessGate}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Observability Dashboard Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Dashboard UX/runtime #oshun-web Platform on-call
Signal and SLO ingestion #oshun-data-platform Data platform lead
Dependency graph/drilldown services #oshun-platform-sre SRE lead
Policy/CAB governance controls #oshun-security Security + Compliance
Alert triage and readiness operations #oshun-program-ops Program Ops leadership

7. Acceptance Criteria (OST-00232)#

OST-00232 is complete only when:

  1. Ownership and decision authority are documented.
  2. Observability/dashboard SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.