Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Data Platform +
Platform SRE + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:
docs/adr/ADR-0042-oshun-studio-experimentation-and-feature-flags.mddocs/releases/v1/specs/studio-experimentation-and-feature-flags-dependency-and-data-contract-map.mddocs/releases/v1/specs/studio-experimentation-and-feature-flags-user-journeys-and-edge-cases.mdapps/oshun/web/src/components/studio/StudioExperimentationFeatureFlagsWorkspace.tsxapps/oshun/web/src/analytics/studioExperimentationFeatureFlagsTelemetry.ts
Purpose#
Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio experimentation and feature-flag workflows.
This document is the execution artifact for OST-00240.
1. Ownership Model#
| Surface | Primary Owner | Backup Owner | Dependencies |
|---|---|---|---|
| Experimentation workspace route and UX | OSHUN Web | Platform Web | route contracts + workspace orchestration |
| Flag definition and lifecycle registry | OSHUN BFF + Platform | Data Platform | canonical definition contracts + owner/expiry governance |
| Targeting/cohort assignment and rollout control | BFF + Data Platform | Platform SRE | bucketing services + release-channel policy engines |
| Outcome evaluation and decision governance | Data Platform + Program | Compliance | exposure/outcome pipelines + confidence evaluation services |
| Rollback/kill-switch safety controls | Security + Platform SRE | Program Operations | runtime control plane + incident orchestration |
| Telemetry/audit continuity | Platform SRE | Data Platform | analytics sinks + audit exports + retention policies |
2. Core SLOs#
| Metric | Target | Alert Threshold |
|---|---|---|
| Experimentation flow success rate | >= 99.0% | < 98.0% over 15 min |
| Flag-definition publish integrity | >= 99.5% | < 99.0% over 15 min |
| Targeting and rollout execution p95 latency | <= 2500ms | > 3500ms over 15 min |
| Exposure/outcome attribution continuity | >= 99.5% | < 99.0% over 15 min |
| Policy-tier and approval enforcement correctness | >= 99.5% | < 99.0% over 15 min |
| Rollback/kill-switch execution determinism | >= 99.5% | < 99.0% over 15 min |
| Telemetry/audit event delivery continuity | >= 99.5% | < 99.0% over 15 min |
3. Operational Playbooks#
3.1 Flag Definition Publish Failures#
Trigger: repeated FLAG_DEFINITION_PUBLISH_FAILED outcomes.
Actions:
- Validate
flagKey, owner, risk class, and expiry metadata fields. - Inspect registry schema compatibility and version-drift diagnostics.
- Re-run publish in preview channel with contract validation enabled.
- Block production rollout operations until publish integrity recovers.
3.2 Targeting and Cohort Configuration Failures#
Trigger: repeated TARGETING_CONFIGURATION_FAILED outcomes.
Actions:
- Validate targeting predicate syntax and deterministic bucket fields.
- Verify variant allocations total 100 and cohort semantics are complete.
- Re-run configuration against known-good contract fixtures.
- Freeze rollout progression for impacted experiment keys until corrected.
3.3 Progressive Rollout Execution Failures#
Trigger: repeated PROGRESSIVE_ROLLOUT_FAILED outcomes.
Actions:
- Validate rollout percentage bounds and release-channel controls.
- Inspect role/tier/CAB policy decisions for deny-path anomalies.
- Re-run rollout in staging with security reviewer oversight.
- Hold production promotion until deterministic rollout execution is restored.
3.4 Outcome Evaluation and Decision Failures#
Trigger: repeated OUTCOME_EVALUATION_FAILED outcomes.
Actions:
- Validate exposure completeness and confidence-score contract fields.
- Inspect KPI aggregation windows and attribution joins for drift.
- Re-run evaluation with compliance auditor supervision.
- Prevent promotion decisions until evidence quality recovers.
3.5 Rollback and Kill-Switch Failures#
Trigger: repeated ROLLBACK_KILL_SWITCH_FAILED outcomes.
Actions:
- Validate selected flag/experiment linkage and active-state prerequisites.
- Inspect runtime propagation path and control-plane acknowledgements.
- Re-run rollback with incident command and restricted tier controls.
- Escalate to Platform SRE if propagation latency exceeds containment SLO.
3.6 Policy Denial Spike#
Trigger: sustained increase in ROLE_POLICY_DENIED,
PERMISSION_TIER_ROLE_DENIED, FLOW_PERMISSION_TIER_INSUFFICIENT,
POLICY_JUSTIFICATION_REQUIRED, HIGH_RISK_DOMAIN_RESTRICTED_TIER_REQUIRED,
PRODUCTION_APPROVAL_REQUIRED, or
PRODUCTION_EVALUATION_REQUIRES_COMPLIANCE_AUDITOR.
Actions:
- Compare active role/tier and production-approval bundles with intended policy.
- Inspect recent policy release deltas for unintended deny-path drift.
- Roll back or patch policy bundle and revalidate allow/deny matrix.
- Confirm denial-rate normalization across all experimentation flows.
3.7 Telemetry and Audit Degradation#
Trigger: missing or delayed studio_experimentation_feature_flags_* telemetry
events.
Actions:
- Validate analytics sink health, backlog depth, and retry posture.
- Enable degraded-observability notice in experimentation workspace operations.
- Replay buffered telemetry where replay semantics are supported.
- Verify SLO panel and governance audit stream completeness after recovery.
4. Release Checklist#
- Validate
/studio/experimentation-feature-flagsrenders and executes all required flows. - Validate optimistic, error, and retry behavior for all experimentation flows.
- Validate permission-tier, high-risk domain, and production CAB policy enforcement.
- Validate telemetry/audit emissions and SLO panel updates.
- Validate integration/e2e/accessibility/performance evidence artifacts.
5. Incident Communication Templates#
Incident Start#
[P{severity}] Studio Experimentation/Feature-Flags Incident
Detected: {timestamp}
Impact: {flag-publish | targeting | rollout | outcome-evaluation | rollback-kill-switch}
Suspected cause: {policy drift | schema regression | telemetry degradation | runtime propagation failure}
Owner: {on-call}
Incident Update#
[P{severity}] Studio Experimentation/Feature-Flags Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {publishFeatureFlagDefinition | configureTargetingCohorts | executeProgressiveRollout | evaluateExperimentOutcomes | triggerRollbackKillSwitch}
ETA: {time}
Incident Resolved#
[P{severity}] Studio Experimentation/Feature-Flags Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
6. Escalation Paths#
| Area | Primary Channel | Escalate To |
|---|---|---|
| Web experimentation workspace UX/runtime | #oshun-web |
Platform on-call |
| Flag lifecycle and rollout control-plane | #oshun-bff |
BFF lead |
| Exposure/outcome analytics and attribution | #oshun-data-platform |
Data platform lead |
| Policy-tier and CAB governance enforcement | #oshun-security |
Security + Compliance |
| Rollback/kill-switch runtime operations | #oshun-platform-sre |
SRE lead |
7. Acceptance Criteria (OST-00240)#
OST-00240 is complete only when:
- Ownership and decision authority are documented.
- Experimentation/feature-flag SLOs and alert thresholds are documented.
- At least five playbooks are documented for major failure modes.
- Release checklist, communication templates, and escalation paths exist.