# OSHUN Studio Experimentation and Feature Flags Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Data Platform +
Platform SRE + OSHUN Security + Program Operations  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0042-oshun-studio-experimentation-and-feature-flags.md`
- `docs/releases/v1/specs/studio-experimentation-and-feature-flags-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-experimentation-and-feature-flags-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioExperimentationFeatureFlagsWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioExperimentationFeatureFlagsTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio experimentation and feature-flag workflows.

This document is the execution artifact for `OST-00240`.

## 1. Ownership Model

| Surface                                         | Primary Owner           | Backup Owner       | Dependencies                                                |
| ----------------------------------------------- | ----------------------- | ------------------ | ----------------------------------------------------------- |
| Experimentation workspace route and UX          | OSHUN Web               | Platform Web       | route contracts + workspace orchestration                   |
| Flag definition and lifecycle registry          | OSHUN BFF + Platform    | Data Platform      | canonical definition contracts + owner/expiry governance    |
| Targeting/cohort assignment and rollout control | BFF + Data Platform     | Platform SRE       | bucketing services + release-channel policy engines         |
| Outcome evaluation and decision governance      | Data Platform + Program | Compliance         | exposure/outcome pipelines + confidence evaluation services |
| Rollback/kill-switch safety controls            | Security + Platform SRE | Program Operations | runtime control plane + incident orchestration              |
| Telemetry/audit continuity                      | Platform SRE            | Data Platform      | analytics sinks + audit exports + retention policies        |

## 2. Core SLOs

| Metric                                           | Target    | Alert Threshold      |
| ------------------------------------------------ | --------- | -------------------- |
| Experimentation flow success rate                | >= 99.0%  | < 98.0% over 15 min  |
| Flag-definition publish integrity                | >= 99.5%  | < 99.0% over 15 min  |
| Targeting and rollout execution p95 latency      | <= 2500ms | > 3500ms over 15 min |
| Exposure/outcome attribution continuity          | >= 99.5%  | < 99.0% over 15 min  |
| Policy-tier and approval enforcement correctness | >= 99.5%  | < 99.0% over 15 min  |
| Rollback/kill-switch execution determinism       | >= 99.5%  | < 99.0% over 15 min  |
| Telemetry/audit event delivery continuity        | >= 99.5%  | < 99.0% over 15 min  |

## 3. Operational Playbooks

### 3.1 Flag Definition Publish Failures

Trigger: repeated `FLAG_DEFINITION_PUBLISH_FAILED` outcomes.

Actions:

1. Validate `flagKey`, owner, risk class, and expiry metadata fields.
2. Inspect registry schema compatibility and version-drift diagnostics.
3. Re-run publish in preview channel with contract validation enabled.
4. Block production rollout operations until publish integrity recovers.

### 3.2 Targeting and Cohort Configuration Failures

Trigger: repeated `TARGETING_CONFIGURATION_FAILED` outcomes.

Actions:

1. Validate targeting predicate syntax and deterministic bucket fields.
2. Verify variant allocations total 100 and cohort semantics are complete.
3. Re-run configuration against known-good contract fixtures.
4. Freeze rollout progression for impacted experiment keys until corrected.

### 3.3 Progressive Rollout Execution Failures

Trigger: repeated `PROGRESSIVE_ROLLOUT_FAILED` outcomes.

Actions:

1. Validate rollout percentage bounds and release-channel controls.
2. Inspect role/tier/CAB policy decisions for deny-path anomalies.
3. Re-run rollout in staging with security reviewer oversight.
4. Hold production promotion until deterministic rollout execution is restored.

### 3.4 Outcome Evaluation and Decision Failures

Trigger: repeated `OUTCOME_EVALUATION_FAILED` outcomes.

Actions:

1. Validate exposure completeness and confidence-score contract fields.
2. Inspect KPI aggregation windows and attribution joins for drift.
3. Re-run evaluation with compliance auditor supervision.
4. Prevent promotion decisions until evidence quality recovers.

### 3.5 Rollback and Kill-Switch Failures

Trigger: repeated `ROLLBACK_KILL_SWITCH_FAILED` outcomes.

Actions:

1. Validate selected flag/experiment linkage and active-state prerequisites.
2. Inspect runtime propagation path and control-plane acknowledgements.
3. Re-run rollback with incident command and restricted tier controls.
4. Escalate to Platform SRE if propagation latency exceeds containment SLO.

### 3.6 Policy Denial Spike

Trigger: sustained increase in `ROLE_POLICY_DENIED`,
`PERMISSION_TIER_ROLE_DENIED`, `FLOW_PERMISSION_TIER_INSUFFICIENT`,
`POLICY_JUSTIFICATION_REQUIRED`, `HIGH_RISK_DOMAIN_RESTRICTED_TIER_REQUIRED`,
`PRODUCTION_APPROVAL_REQUIRED`, or
`PRODUCTION_EVALUATION_REQUIRES_COMPLIANCE_AUDITOR`.

Actions:

1. Compare active role/tier and production-approval bundles with intended
   policy.
2. Inspect recent policy release deltas for unintended deny-path drift.
3. Roll back or patch policy bundle and revalidate allow/deny matrix.
4. Confirm denial-rate normalization across all experimentation flows.

### 3.7 Telemetry and Audit Degradation

Trigger: missing or delayed `studio_experimentation_feature_flags_*` telemetry
events.

Actions:

1. Validate analytics sink health, backlog depth, and retry posture.
2. Enable degraded-observability notice in experimentation workspace operations.
3. Replay buffered telemetry where replay semantics are supported.
4. Verify SLO panel and governance audit stream completeness after recovery.

## 4. Release Checklist

- [ ] Validate `/studio/experimentation-feature-flags` renders and executes all
      required flows.
- [ ] Validate optimistic, error, and retry behavior for all experimentation
      flows.
- [ ] Validate permission-tier, high-risk domain, and production CAB policy
      enforcement.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility/performance evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```
[P{severity}] Studio Experimentation/Feature-Flags Incident
Detected: {timestamp}
Impact: {flag-publish | targeting | rollout | outcome-evaluation | rollback-kill-switch}
Suspected cause: {policy drift | schema regression | telemetry degradation | runtime propagation failure}
Owner: {on-call}
```

### Incident Update

```
[P{severity}] Studio Experimentation/Feature-Flags Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {publishFeatureFlagDefinition | configureTargetingCohorts | executeProgressiveRollout | evaluateExperimentOutcomes | triggerRollbackKillSwitch}
ETA: {time}
```

### Incident Resolved

```
[P{severity}] Studio Experimentation/Feature-Flags Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                                       | Primary Channel        | Escalate To           |
| ------------------------------------------ | ---------------------- | --------------------- |
| Web experimentation workspace UX/runtime   | `#oshun-web`           | Platform on-call      |
| Flag lifecycle and rollout control-plane   | `#oshun-bff`           | BFF lead              |
| Exposure/outcome analytics and attribution | `#oshun-data-platform` | Data platform lead    |
| Policy-tier and CAB governance enforcement | `#oshun-security`      | Security + Compliance |
| Rollback/kill-switch runtime operations    | `#oshun-platform-sre`  | SRE lead              |

## 7. Acceptance Criteria (OST-00240)

`OST-00240` is complete only when:

1. Ownership and decision authority are documented.
2. Experimentation/feature-flag SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
