# OSHUN Studio Metrics and Analytics Instrumentation Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Data Platform +
Platform SRE + OSHUN Security + Program Operations  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0040-oshun-studio-metrics-and-analytics-instrumentation.md`
- `docs/releases/v1/specs/studio-metrics-and-analytics-instrumentation-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-metrics-and-analytics-instrumentation-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioMetricsAnalyticsInstrumentationWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioMetricsAnalyticsInstrumentationTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio metrics and analytics instrumentation workflows.

This document is the execution artifact for `OST-00224`.

## 1. Ownership Model

| Surface                                       | Primary Owner             | Backup Owner       | Dependencies                                           |
| --------------------------------------------- | ------------------------- | ------------------ | ------------------------------------------------------ |
| Metrics instrumentation workspace route       | OSHUN Web                 | Platform Web       | route contracts + instrumentation UX orchestration     |
| Metric definition publishing and lineage      | OSHUN BFF + Data Platform | Platform SRE       | schema registry + entity/ownership contracts           |
| Flow-span and KPI evaluation orchestration    | Data Platform             | BFF                | analytics sinks + threshold evaluation services        |
| Permission-tier and CAB release policy checks | Security + Compliance     | Program Operations | RBAC/policy bundles + governance approval systems      |
| Audit evidence and telemetry quality gating   | Platform SRE + Compliance | Data Platform      | telemetry ingestion + quality verifiers + audit export |

## 2. Core SLOs

| Metric                                              | Target    | Alert Threshold      |
| --------------------------------------------------- | --------- | -------------------- |
| Metrics instrumentation flow success rate           | >= 99.0%  | < 98.0% over 15 min  |
| Metric-definition publish integrity                 | >= 99.5%  | < 99.0% over 15 min  |
| Metrics instrumentation flow p95 latency            | <= 2500ms | > 3500ms over 15 min |
| Audit-linkage completeness and canonicality         | >= 99.5%  | < 99.0% over 15 min  |
| Release-gate verdict determinism                    | >= 99.5%  | < 99.0% over 15 min  |
| Telemetry quality/audit evidence continuity         | >= 99.5%  | < 99.0% over 15 min  |
| Permission-tier and policy-denial anomaly detection | >= 99.0%  | < 98.0% over 15 min  |

## 3. Operational Playbooks

### 3.1 Metric Definition Publish Failures

Trigger: repeated `METRIC_DEFINITION_PUBLISH_FAILED` outcomes.

Actions:

1. Validate metric identity (`metricId`, `metricName`, `owner`) and threshold
   metadata.
2. Inspect schema/contract registry compatibility for publish payload drift.
3. Re-run publish in preview channel with contract diagnostics enabled.
4. Block production promotion until definition integrity stabilizes.

### 3.2 Flow Span Emission Degradation

Trigger: repeated `FLOW_SPAN_EMISSION_FAILED` outcomes.

Actions:

1. Validate requested/completed span pair payload and flow attribution fields.
2. Inspect analytics sink backpressure and span lifecycle ordering.
3. Re-run span emission with tracing enabled and bounded traffic replay.
4. Escalate to Platform SRE if p95 latency or completion gaps persist.

### 3.3 Audit-Linkage Contract Failures

Trigger: repeated `AUDIT_ANALYTICS_LINKAGE_FAILED` outcomes.

Actions:

1. Validate canonical audit event name and lineage mapping fields.
2. Inspect governance mapping services for version or schema drift.
3. Re-run linkage in staging with compliance observer sign-off.
4. Freeze production linkage updates until CAB-approved remediation is applied.

### 3.4 KPI Evaluation and Threshold Contract Failures

Trigger: repeated `KPI_EVALUATION_FAILED` outcomes.

Actions:

1. Validate warning/critical thresholds and KPI composition inputs.
2. Inspect threshold evaluation engine for rounding or window drift.
3. Re-run KPI evaluation against known-good fixtures.
4. Disable promotion gates that depend on invalid KPI verdicts until fixed.

### 3.5 Telemetry Quality Release-Gate Failures

Trigger: repeated `TELEMETRY_QUALITY_GATE_FAILED` outcomes.

Actions:

1. Validate coverage, linkage, and quality-state prerequisites.
2. Inspect completeness and schema-violation streams for regression sources.
3. Re-run release gate with compliance auditor supervision.
4. Hold production release path until gate returns deterministic verdict.

### 3.6 Permission and Policy Denial Spike

Trigger: sustained increase in `ROLE_POLICY_DENIED`,
`PERMISSION_TIER_ROLE_DENIED`, `FLOW_PERMISSION_TIER_INSUFFICIENT`,
`POLICY_JUSTIFICATION_REQUIRED`, `RESTRICTED_DOMAIN_REQUIRES_RESTRICTED_TIER`,
`PRODUCTION_APPROVAL_TICKET_REQUIRED`, or
`PRODUCTION_RELEASE_GATE_REQUIRES_COMPLIANCE_AUDITOR`.

Actions:

1. Compare active role/permission-tier policies with intended policy bundles.
2. Inspect recent policy release deltas for unintended deny-path regressions.
3. Roll back or patch policy bundle and verify expected allow/deny behavior.
4. Confirm deny-rate normalization and recovery across all instrumentation
   flows.

### 3.7 Telemetry and Audit Evidence Degradation

Trigger: missing or delayed `studio_metrics_analytics_instrumentation_*`
telemetry events.

Actions:

1. Validate analytics sink health and backlog depth.
2. Enable degraded-observability banner in metrics workspace operations.
3. Replay buffered telemetry where replay semantics are supported.
4. Verify SLO dashboards and governance evidence stream completeness.

## 4. Release Checklist

- [ ] Validate `/studio/metrics-analytics-instrumentation` renders and executes
      required flows.
- [ ] Validate optimistic, error, and retry behavior for all instrumentation
      flows.
- [ ] Validate permission-tier, restricted-domain, and production CAB policy
      enforcement.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility/performance evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```
[P{severity}] Studio Metrics/Analytics Instrumentation Incident
Detected: {timestamp}
Impact: {metric-publish | span-emission | audit-linkage | kpi-evaluation | release-gate}
Suspected cause: {policy drift | schema regression | telemetry degradation | threshold contract drift}
Owner: {on-call}
```

### Incident Update

```
[P{severity}] Studio Metrics/Analytics Instrumentation Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {publishMetricDefinition | emitFlowSpanPair | linkAuditAnalyticsOutcome | composeKpiThresholdEvaluation | runTelemetryQualityReleaseGate}
ETA: {time}
```

### Incident Resolved

```
[P{severity}] Studio Metrics/Analytics Instrumentation Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                                    | Primary Channel        | Escalate To            |
| --------------------------------------- | ---------------------- | ---------------------- |
| Web runtime and instrumentation UX      | `#oshun-web`           | Platform on-call       |
| Metrics contracts and BFF APIs          | `#oshun-bff`           | BFF lead               |
| Telemetry ingestion and quality gating  | `#oshun-platform-sre`  | SRE lead               |
| Policy/entitlement/CAB governance rules | `#oshun-security`      | Security + Compliance  |
| Analytics evidence and reporting        | `#oshun-data-platform` | Data platform/SRE lead |

## 7. Acceptance Criteria (OST-00224)

`OST-00224` is complete only when:

1. Ownership and decision authority are documented.
2. Metrics/analytics instrumentation SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
