Eve library

OSHUN Studio Performance Budgets Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN Data/Analytics + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0043-oshun-studio-performance-budgets.md
  • docs/releases/v1/specs/studio-performance-budgets-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-performance-budgets-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioPerformanceBudgetsWorkspace.tsx
  • apps/oshun/web/src/analytics/studioPerformanceBudgetsTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio performance-budget workflows.

This document is the execution artifact for OST-00248.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Performance budgets workspace route OSHUN Web Platform Web route contracts + budget UX orchestration
Budget definition and lifecycle contracts OSHUN BFF + Platform Data Platform schema contracts + owner/expiry governance
CI/runtime budget validation pipeline Platform SRE Data Platform web-vitals + bundle/perf tests + validation artifacts
Waiver and release-block policy controls Security + Compliance Program Operations RBAC/tier policy bundles + CAB governance
Breach remediation and closure orchestration Platform SRE + Security Domain service ops incident workflows + rollback/hotfix paths
Telemetry and audit evidence continuity Data Platform + SRE Security analytics ingestion + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
Performance budget flow success rate >= 99.0% < 98.0% over 15 min
Budget definition integrity >= 99.5% < 99.0% over 15 min
Validation and remediation flow p95 latency <= 2500ms > 3500ms over 15 min
Runtime budget measurement continuity >= 99.5% < 99.0% over 15 min
Policy-tier/CAB enforcement correctness >= 99.5% < 99.0% over 15 min
Tier-1 critical breach handling determinism >= 99.5% < 99.0% over 15 min
Telemetry/audit event delivery continuity >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Budget Definition Publish Failures#

Trigger: repeated PERFORMANCE_BUDGET_PUBLISH_FAILED outcomes.

Actions:

  1. Validate budget identity (budgetId, budgetKey, metric, threshold) and owner metadata.
  2. Inspect contract registry compatibility for schema/version drift.
  3. Re-run publish in preview channel with contract diagnostics enabled.
  4. Block production waiver/remediation actions until definition integrity recovers.

3.2 CI Budget Validation Failures#

Trigger: repeated PERFORMANCE_BUDGET_CI_VALIDATION_FAILED outcomes.

Actions:

  1. Validate failing metric evidence and artifact references.
  2. Inspect CI budget assertion suites for regression/fixture drift.
  3. Re-run validation against known-good baseline artifacts.
  4. Keep release blocked until deterministic pass criteria are restored.

3.3 Runtime Posture Monitoring Degradation#

Trigger: repeated PERFORMANCE_BUDGET_RUNTIME_MONITORING_FAILED outcomes.

Actions:

  1. Validate runtime telemetry ingestion and budget binding identity.
  2. Inspect lag/backlog for measurement stream degradation.
  3. Re-run runtime posture checks with bounded diagnostic windows.
  4. Escalate to Platform SRE for sustained tier-1 posture regressions.

3.4 Waiver/Release-Block Decision Failures#

Trigger: repeated PERFORMANCE_BUDGET_POLICY_DECISION_FAILED outcomes.

Actions:

  1. Validate role/tier/CAB decision prerequisites and reason codes.
  2. Inspect policy bundle drift for deny-path anomalies.
  3. Re-run decision with compliance auditor supervision.
  4. Freeze production promotions while policy integrity is degraded.

3.5 Remediation Closure Failures#

Trigger: repeated PERFORMANCE_BUDGET_REMEDIATION_CLOSURE_FAILED outcomes.

Actions:

  1. Validate incident-state and remediation-action preconditions.
  2. Inspect post-fix validation evidence and rollback/hotfix artifacts.
  3. Re-run remediation closure with incident command oversight.
  4. Prevent incident closure until revalidation passes and evidence is complete.

3.6 Policy Denial Spike#

Trigger: sustained increase in ROLE_POLICY_DENIED, PERMISSION_TIER_ROLE_DENIED, FLOW_PERMISSION_TIER_INSUFFICIENT, POLICY_JUSTIFICATION_REQUIRED, TIER_1_CRITICAL_RESTRICTED_REQUIRED, PRODUCTION_APPROVAL_REQUIRED, or PRODUCTION_WAIVER_REQUIRES_COMPLIANCE_AUDITOR.

Actions:

  1. Compare active role/tier/CAB policy bundles against intended controls.
  2. Inspect recent policy releases for deny-path drift.
  3. Roll back or patch policy bundle and revalidate decision paths.
  4. Confirm deny-rate normalization across all performance-budget flows.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed studio_performance_budgets_* telemetry events.

Actions:

  1. Validate analytics sink health and backlog depth.
  2. Enable degraded-observability notice in performance budgets workspace.
  3. Replay buffered telemetry where replay semantics are supported.
  4. Confirm SLO panel and audit evidence continuity after recovery.

4. Release Checklist#

  • Validate /studio/performance-budgets renders and executes required flows.
  • Validate optimistic, error, and retry behavior for all budget flows.
  • Validate permission-tier, tier-1 critical, and production CAB policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility/performance evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Performance Budget Incident
Detected: {timestamp}
Impact: {budget-publish | ci-validation | runtime-monitoring | policy-decision | remediation-closure}
Suspected cause: {policy drift | schema regression | telemetry degradation | runtime regression}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Performance Budget Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {publishPerformanceBudgetDefinition | runCiBudgetValidation | monitorRuntimeBudgetPosture | decideWaiverOrReleaseBlock | executeBreachRemediationClosure}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Performance Budget Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Performance budget workspace UX #oshun-web Platform on-call
Budget contracts and validation APIs #oshun-bff BFF lead
Runtime telemetry and posture services #oshun-platform-sre SRE lead
Policy-tier/CAB governance controls #oshun-security Security + Compliance
Budget analytics and reporting #oshun-data-platform Data platform lead

7. Acceptance Criteria (OST-00248)#

OST-00248 is complete only when:

  1. Ownership and decision authority are documented.
  2. Performance-budget SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.