Eve library

OSHUN Studio Experimentation and Feature Flags Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Data Platform + Platform SRE + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0042-oshun-studio-experimentation-and-feature-flags.md
  • docs/releases/v1/specs/studio-experimentation-and-feature-flags-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-experimentation-and-feature-flags-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioExperimentationFeatureFlagsWorkspace.tsx
  • apps/oshun/web/src/analytics/studioExperimentationFeatureFlagsTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio experimentation and feature-flag workflows.

This document is the execution artifact for OST-00240.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Experimentation workspace route and UX OSHUN Web Platform Web route contracts + workspace orchestration
Flag definition and lifecycle registry OSHUN BFF + Platform Data Platform canonical definition contracts + owner/expiry governance
Targeting/cohort assignment and rollout control BFF + Data Platform Platform SRE bucketing services + release-channel policy engines
Outcome evaluation and decision governance Data Platform + Program Compliance exposure/outcome pipelines + confidence evaluation services
Rollback/kill-switch safety controls Security + Platform SRE Program Operations runtime control plane + incident orchestration
Telemetry/audit continuity Platform SRE Data Platform analytics sinks + audit exports + retention policies

2. Core SLOs#

Metric Target Alert Threshold
Experimentation flow success rate >= 99.0% < 98.0% over 15 min
Flag-definition publish integrity >= 99.5% < 99.0% over 15 min
Targeting and rollout execution p95 latency <= 2500ms > 3500ms over 15 min
Exposure/outcome attribution continuity >= 99.5% < 99.0% over 15 min
Policy-tier and approval enforcement correctness >= 99.5% < 99.0% over 15 min
Rollback/kill-switch execution determinism >= 99.5% < 99.0% over 15 min
Telemetry/audit event delivery continuity >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Flag Definition Publish Failures#

Trigger: repeated FLAG_DEFINITION_PUBLISH_FAILED outcomes.

Actions:

  1. Validate flagKey, owner, risk class, and expiry metadata fields.
  2. Inspect registry schema compatibility and version-drift diagnostics.
  3. Re-run publish in preview channel with contract validation enabled.
  4. Block production rollout operations until publish integrity recovers.

3.2 Targeting and Cohort Configuration Failures#

Trigger: repeated TARGETING_CONFIGURATION_FAILED outcomes.

Actions:

  1. Validate targeting predicate syntax and deterministic bucket fields.
  2. Verify variant allocations total 100 and cohort semantics are complete.
  3. Re-run configuration against known-good contract fixtures.
  4. Freeze rollout progression for impacted experiment keys until corrected.

3.3 Progressive Rollout Execution Failures#

Trigger: repeated PROGRESSIVE_ROLLOUT_FAILED outcomes.

Actions:

  1. Validate rollout percentage bounds and release-channel controls.
  2. Inspect role/tier/CAB policy decisions for deny-path anomalies.
  3. Re-run rollout in staging with security reviewer oversight.
  4. Hold production promotion until deterministic rollout execution is restored.

3.4 Outcome Evaluation and Decision Failures#

Trigger: repeated OUTCOME_EVALUATION_FAILED outcomes.

Actions:

  1. Validate exposure completeness and confidence-score contract fields.
  2. Inspect KPI aggregation windows and attribution joins for drift.
  3. Re-run evaluation with compliance auditor supervision.
  4. Prevent promotion decisions until evidence quality recovers.

3.5 Rollback and Kill-Switch Failures#

Trigger: repeated ROLLBACK_KILL_SWITCH_FAILED outcomes.

Actions:

  1. Validate selected flag/experiment linkage and active-state prerequisites.
  2. Inspect runtime propagation path and control-plane acknowledgements.
  3. Re-run rollback with incident command and restricted tier controls.
  4. Escalate to Platform SRE if propagation latency exceeds containment SLO.

3.6 Policy Denial Spike#

Trigger: sustained increase in ROLE_POLICY_DENIED, PERMISSION_TIER_ROLE_DENIED, FLOW_PERMISSION_TIER_INSUFFICIENT, POLICY_JUSTIFICATION_REQUIRED, HIGH_RISK_DOMAIN_RESTRICTED_TIER_REQUIRED, PRODUCTION_APPROVAL_REQUIRED, or PRODUCTION_EVALUATION_REQUIRES_COMPLIANCE_AUDITOR.

Actions:

  1. Compare active role/tier and production-approval bundles with intended policy.
  2. Inspect recent policy release deltas for unintended deny-path drift.
  3. Roll back or patch policy bundle and revalidate allow/deny matrix.
  4. Confirm denial-rate normalization across all experimentation flows.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed studio_experimentation_feature_flags_* telemetry events.

Actions:

  1. Validate analytics sink health, backlog depth, and retry posture.
  2. Enable degraded-observability notice in experimentation workspace operations.
  3. Replay buffered telemetry where replay semantics are supported.
  4. Verify SLO panel and governance audit stream completeness after recovery.

4. Release Checklist#

  • Validate /studio/experimentation-feature-flags renders and executes all required flows.
  • Validate optimistic, error, and retry behavior for all experimentation flows.
  • Validate permission-tier, high-risk domain, and production CAB policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility/performance evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Experimentation/Feature-Flags Incident
Detected: {timestamp}
Impact: {flag-publish | targeting | rollout | outcome-evaluation | rollback-kill-switch}
Suspected cause: {policy drift | schema regression | telemetry degradation | runtime propagation failure}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Experimentation/Feature-Flags Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {publishFeatureFlagDefinition | configureTargetingCohorts | executeProgressiveRollout | evaluateExperimentOutcomes | triggerRollbackKillSwitch}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Experimentation/Feature-Flags Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web experimentation workspace UX/runtime #oshun-web Platform on-call
Flag lifecycle and rollout control-plane #oshun-bff BFF lead
Exposure/outcome analytics and attribution #oshun-data-platform Data platform lead
Policy-tier and CAB governance enforcement #oshun-security Security + Compliance
Rollback/kill-switch runtime operations #oshun-platform-sre SRE lead

7. Acceptance Criteria (OST-00240)#

OST-00240 is complete only when:

  1. Ownership and decision authority are documented.
  2. Experimentation/feature-flag SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.