Eve library

OSHUN Studio Metrics and Analytics Instrumentation Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Data Platform + Platform SRE + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0040-oshun-studio-metrics-and-analytics-instrumentation.md
  • docs/releases/v1/specs/studio-metrics-and-analytics-instrumentation-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-metrics-and-analytics-instrumentation-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioMetricsAnalyticsInstrumentationWorkspace.tsx
  • apps/oshun/web/src/analytics/studioMetricsAnalyticsInstrumentationTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio metrics and analytics instrumentation workflows.

This document is the execution artifact for OST-00224.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Metrics instrumentation workspace route OSHUN Web Platform Web route contracts + instrumentation UX orchestration
Metric definition publishing and lineage OSHUN BFF + Data Platform Platform SRE schema registry + entity/ownership contracts
Flow-span and KPI evaluation orchestration Data Platform BFF analytics sinks + threshold evaluation services
Permission-tier and CAB release policy checks Security + Compliance Program Operations RBAC/policy bundles + governance approval systems
Audit evidence and telemetry quality gating Platform SRE + Compliance Data Platform telemetry ingestion + quality verifiers + audit export

2. Core SLOs#

Metric Target Alert Threshold
Metrics instrumentation flow success rate >= 99.0% < 98.0% over 15 min
Metric-definition publish integrity >= 99.5% < 99.0% over 15 min
Metrics instrumentation flow p95 latency <= 2500ms > 3500ms over 15 min
Audit-linkage completeness and canonicality >= 99.5% < 99.0% over 15 min
Release-gate verdict determinism >= 99.5% < 99.0% over 15 min
Telemetry quality/audit evidence continuity >= 99.5% < 99.0% over 15 min
Permission-tier and policy-denial anomaly detection >= 99.0% < 98.0% over 15 min

3. Operational Playbooks#

3.1 Metric Definition Publish Failures#

Trigger: repeated METRIC_DEFINITION_PUBLISH_FAILED outcomes.

Actions:

  1. Validate metric identity (metricId, metricName, owner) and threshold metadata.
  2. Inspect schema/contract registry compatibility for publish payload drift.
  3. Re-run publish in preview channel with contract diagnostics enabled.
  4. Block production promotion until definition integrity stabilizes.

3.2 Flow Span Emission Degradation#

Trigger: repeated FLOW_SPAN_EMISSION_FAILED outcomes.

Actions:

  1. Validate requested/completed span pair payload and flow attribution fields.
  2. Inspect analytics sink backpressure and span lifecycle ordering.
  3. Re-run span emission with tracing enabled and bounded traffic replay.
  4. Escalate to Platform SRE if p95 latency or completion gaps persist.

3.3 Audit-Linkage Contract Failures#

Trigger: repeated AUDIT_ANALYTICS_LINKAGE_FAILED outcomes.

Actions:

  1. Validate canonical audit event name and lineage mapping fields.
  2. Inspect governance mapping services for version or schema drift.
  3. Re-run linkage in staging with compliance observer sign-off.
  4. Freeze production linkage updates until CAB-approved remediation is applied.

3.4 KPI Evaluation and Threshold Contract Failures#

Trigger: repeated KPI_EVALUATION_FAILED outcomes.

Actions:

  1. Validate warning/critical thresholds and KPI composition inputs.
  2. Inspect threshold evaluation engine for rounding or window drift.
  3. Re-run KPI evaluation against known-good fixtures.
  4. Disable promotion gates that depend on invalid KPI verdicts until fixed.

3.5 Telemetry Quality Release-Gate Failures#

Trigger: repeated TELEMETRY_QUALITY_GATE_FAILED outcomes.

Actions:

  1. Validate coverage, linkage, and quality-state prerequisites.
  2. Inspect completeness and schema-violation streams for regression sources.
  3. Re-run release gate with compliance auditor supervision.
  4. Hold production release path until gate returns deterministic verdict.

3.6 Permission and Policy Denial Spike#

Trigger: sustained increase in ROLE_POLICY_DENIED, PERMISSION_TIER_ROLE_DENIED, FLOW_PERMISSION_TIER_INSUFFICIENT, POLICY_JUSTIFICATION_REQUIRED, RESTRICTED_DOMAIN_REQUIRES_RESTRICTED_TIER, PRODUCTION_APPROVAL_TICKET_REQUIRED, or PRODUCTION_RELEASE_GATE_REQUIRES_COMPLIANCE_AUDITOR.

Actions:

  1. Compare active role/permission-tier policies with intended policy bundles.
  2. Inspect recent policy release deltas for unintended deny-path regressions.
  3. Roll back or patch policy bundle and verify expected allow/deny behavior.
  4. Confirm deny-rate normalization and recovery across all instrumentation flows.

3.7 Telemetry and Audit Evidence Degradation#

Trigger: missing or delayed studio_metrics_analytics_instrumentation_* telemetry events.

Actions:

  1. Validate analytics sink health and backlog depth.
  2. Enable degraded-observability banner in metrics workspace operations.
  3. Replay buffered telemetry where replay semantics are supported.
  4. Verify SLO dashboards and governance evidence stream completeness.

4. Release Checklist#

  • Validate /studio/metrics-analytics-instrumentation renders and executes required flows.
  • Validate optimistic, error, and retry behavior for all instrumentation flows.
  • Validate permission-tier, restricted-domain, and production CAB policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility/performance evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Metrics/Analytics Instrumentation Incident
Detected: {timestamp}
Impact: {metric-publish | span-emission | audit-linkage | kpi-evaluation | release-gate}
Suspected cause: {policy drift | schema regression | telemetry degradation | threshold contract drift}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Metrics/Analytics Instrumentation Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {publishMetricDefinition | emitFlowSpanPair | linkAuditAnalyticsOutcome | composeKpiThresholdEvaluation | runTelemetryQualityReleaseGate}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Metrics/Analytics Instrumentation Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and instrumentation UX #oshun-web Platform on-call
Metrics contracts and BFF APIs #oshun-bff BFF lead
Telemetry ingestion and quality gating #oshun-platform-sre SRE lead
Policy/entitlement/CAB governance rules #oshun-security Security + Compliance
Analytics evidence and reporting #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00224)#

OST-00224 is complete only when:

  1. Ownership and decision authority are documented.
  2. Metrics/analytics instrumentation SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.