Eve library

OSHUN Studio Review and Approval Workflows Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0035-oshun-studio-review-and-approval-workflows.md
  • docs/releases/v1/specs/studio-review-and-approval-workflows-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-review-and-approval-workflows-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioReviewApprovalWorkflowsWorkspace.tsx
  • apps/oshun/web/src/analytics/studioReviewApprovalWorkflowsTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio review and approval workflows.

This document is the execution artifact for OST-00184.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Review/approval workspace route OSHUN Web Platform Web route contracts + UI orchestration
Review package and stage persistence OSHUN BFF Platform SRE package persistence + stage transition services
Decision lifecycle orchestration Program Operations Studio Architecture decision contracts + notification pipelines
Delegation and escalation policy Security + Compliance Program Operations policy bundles + entitlement services
Governance evidence export pipeline Data Platform + Security Platform SRE telemetry ingestion + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
Review/approval flow success rate >= 99.0% < 98.0% over 15 min
Stage transition reliability >= 99.5% < 99.0% over 15 min
Review/approval flow p95 latency <= 2300ms > 3300ms over 15 min
Decision attribution completeness >= 99.5% < 99.0% over 15 min
Delegation/escalation policy integrity >= 99.5% < 99.0% over 15 min
Review/approval audit event completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Review Package Submit Failures#

Trigger: repeated REVIEW_PACKAGE_SUBMIT_FAILED outcomes.

Actions:

  1. Validate package envelope requirements (reviewId, artifactRef, owner, requestedBy, stageGraphId).
  2. Inspect artifact adapter output for missing dependency or owner metadata.
  3. Re-run submit flow in preview with diagnostics enabled.
  4. Block release promotion until package validation stability normalizes.

3.2 Stage Assignment Failures#

Trigger: repeated STAGE_ASSIGNMENT_FAILED outcomes.

Actions:

  1. Validate stage graph edges and merge semantics.
  2. Inspect stage assignment pipeline for non-deterministic transition behavior.
  3. Re-run stage assignment flow with transition tracing enabled.
  4. Freeze workflow release until deterministic routing is restored.

3.3 Decision Lifecycle Failures#

Trigger: repeated DECISION_LIFECYCLE_FAILED outcomes.

Actions:

  1. Validate decision outcome/rationale payload completeness.
  2. Inspect lifecycle transition logs for illegal state transitions.
  3. Re-run decision flow with decision-history tracing enabled.
  4. Escalate to Studio architecture if lifecycle contract drift persists.

3.4 Delegation/Escalation Routing Failures#

Trigger: repeated DELEGATION_ESCALATION_FAILED outcomes.

Actions:

  1. Validate delegation action, target role, and policy metadata.
  2. Inspect routing and entitlement checks for policy drift.
  3. Re-run delegation/escalation flow with policy trace logging.
  4. Lock routing rollout when authorization checks fail repeatedly.

3.5 Governance Closure Export Failures#

Trigger: repeated GOVERNANCE_CLOSURE_EXPORT_FAILED outcomes.

Actions:

  1. Validate final decision prerequisites and evidence package schema.
  2. Inspect governance export service for retention-tagging issues.
  3. Re-run closure export with compliance reviewer oversight.
  4. Block production closure workflow until export validation passes.

3.6 Policy Denial Spike#

Trigger: unexpected rise in ROLE_POLICY_DENIED, PRODUCTION_APPROVAL_POLICY_DENIED, ESCALATION_AUTHORIZATION_REQUIRED, or ESCALATED_CLOSURE_POLICY_DENIED.

Actions:

  1. Compare flow allowlists with active entitlement and policy bundles.
  2. Inspect recent policy release changes for drift.
  3. Apply policy hotfix or roll back drifted policy bundle.
  4. Confirm deny-rate normalization and impacted workflow recovery.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed review/approval telemetry and audit events.

Actions:

  1. Validate analytics sink health and ingestion backlog depth.
  2. Enable degraded-observability warning in workspace.
  3. Replay buffered events where supported.
  4. Confirm SLO and evidence stream completeness after recovery.

4. Release Checklist#

  • Validate /studio/review-approval-workflows renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate role-policy, escalation, and closure governance enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Review/Approval Workflows Incident
Detected: {timestamp}
Impact: {package-submit | stage-assignment | decision-lifecycle | delegation-routing | closure-export}
Suspected cause: {policy drift | stage graph regression | evidence export failure}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Review/Approval Workflows Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {submitReviewPackage | stageAssignmentOrchestration | decisionLifecycleExecution | delegationEscalationRouting | governanceClosureExport}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Review/Approval Workflows Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and approval workspace #oshun-web Platform on-call
Package/stage persistence services #oshun-bff BFF lead
Workflow transition reliability #oshun-platform-sre SRE lead
Delegation/escalation policy governance #oshun-security Security lead
Audit/compliance evidence pipelines #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00184)#

OST-00184 is complete only when:

  1. Ownership and decision authority are documented.
  2. Review/approval workflow SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.