Eve library

OSHUN Studio API Gateway and BFF Composition Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN Security + OSHUN Data/Analytics Engineering + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0046-oshun-studio-api-gateway-and-bff-composition.md
  • docs/releases/v1/specs/studio-api-gateway-and-bff-composition-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-api-gateway-and-bff-composition-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioApiGatewayBffCompositionWorkspace.tsx
  • apps/oshun/web/src/analytics/studioApiGatewayBffCompositionTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio API gateway and BFF composition workflows.

This document is the execution artifact for OST-00272.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
API gateway/BFF composition workspace route OSHUN Web Platform Web route contracts + composition UX orchestration
Gateway route registry and contract lifecycle OSHUN BFF Studio Architecture schema/version governance + route ownership
Domain adapter composition and aggregation OSHUN BFF + domain teams Platform SRE adapter contracts + transformation pipelines
Policy/entitlement and approval controls Security + Compliance Program Operations role/tier policy bundles + production approval flows
Resilience/fallback and degraded-mode execution Platform SRE + BFF Domain operations timeout/retry/fallback controls + dependency health
Telemetry and governance audit continuity Data Platform + Security Platform SRE analytics sinks + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
Gateway composition flow success rate >= 99.0% < 98.0% over 15 min
Route contract integrity >= 99.5% < 99.0% over 15 min
Composition execution and recovery flow p95 <= 2500ms > 3500ms over 15 min
Downstream dependency health signal continuity >= 99.5% < 99.0% over 15 min
Policy-tier/CAB enforcement correctness >= 99.5% < 99.0% over 15 min
Degraded-mode fallback determinism >= 99.5% < 99.0% over 15 min
Telemetry/audit event delivery continuity >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Route Contract Publish Failures#

Trigger: repeated API_GATEWAY_ROUTE_PUBLISH_FAILED outcomes.

Actions:

  1. Validate route identity (routePath, method, workflowRef, version).
  2. Inspect schema compatibility and contract version drift.
  3. Re-run publish in preview with contract diagnostics enabled.
  4. Block sensitive production actions until route integrity recovers.

3.2 Composed Request Execution Failures#

Trigger: repeated API_GATEWAY_COMPOSITION_EXECUTION_FAILED outcomes.

Actions:

  1. Validate request envelope (requestId, traceId, actor/workspace context).
  2. Inspect adapter fan-out order and downstream error patterns.
  3. Re-run composed request against known-good dependency snapshots.
  4. Escalate to Platform SRE on sustained execution latency/error regressions.

3.3 Degraded-Mode Strategy Failures#

Trigger: repeated API_GATEWAY_DEGRADED_MODE_FAILED outcomes.

Actions:

  1. Validate degraded strategy (fallback_cached, fallback_partial, fallback_minimal) prerequisites.
  2. Inspect dependency health signals and timeout budget drift.
  3. Re-run degradation strategy with bounded fallback diagnostics.
  4. Enforce release hold while degraded-mode determinism is uncertain.

3.4 Sensitive Route Decision Failures#

Trigger: repeated API_GATEWAY_SENSITIVE_ROUTE_DECISION_FAILED outcomes.

Actions:

  1. Validate role/tier/CAB prerequisites and reason codes.
  2. Inspect policy bundle drift and deny-path anomalies.
  3. Re-run decision with compliance auditor supervision.
  4. Freeze production promotion path while policy integrity is degraded.

3.5 Composition Health Recovery Failures#

Trigger: repeated API_GATEWAY_COMPOSITION_RECOVERY_FAILED outcomes.

Actions:

  1. Validate remediation action payload and incident-state preconditions.
  2. Verify post-fix composition validation before closure.
  3. Re-run recovery flow under incident command oversight.
  4. Prevent closure until deterministic revalidation is complete.

3.6 Policy Denial Spike#

Trigger: sustained increase in ROLE_POLICY_DENIED, PERMISSION_TIER_ROLE_DENIED, FLOW_PERMISSION_TIER_INSUFFICIENT, POLICY_JUSTIFICATION_REQUIRED, CRITICAL_ROUTE_RESTRICTED_REQUIRED, PRODUCTION_APPROVAL_REQUIRED, or PRODUCTION_DECISION_REQUIRES_COMPLIANCE_AUDITOR.

Actions:

  1. Compare active role/tier/CAB policy bundles against intended controls.
  2. Inspect recent policy releases for deny-path regressions.
  3. Roll back or patch policy bundle and revalidate decision paths.
  4. Confirm denial-rate normalization across all composition flows.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed studio_api_gateway_bff_composition_* telemetry events.

Actions:

  1. Validate analytics sink health and ingestion backlog.
  2. Enable degraded-observability notice in gateway composition workspace.
  3. Replay buffered events where replay semantics are supported.
  4. Confirm SLO panel and audit-evidence continuity after recovery.

4. Release Checklist#

  • Validate /studio/api-gateway-bff-composition renders and executes required flows.
  • Validate optimistic, error, and retry behavior for all composition flows.
  • Validate permission-tier and production CAB policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility/performance evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio API Gateway/BFF Composition Incident
Detected: {timestamp}
Impact: {route-publish | request-execution | degraded-mode | policy-decision | recovery}
Suspected cause: {policy drift | schema regression | dependency degradation | telemetry degradation}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio API Gateway/BFF Composition Update
Time: {timestamp}
Mitigation: {rollback | hotfix | policy patch | release hold}
Affected flows: {publishGatewayRouteContract | executeComposedRequest | applyDegradedCompositionStrategy | decideSensitiveRouteAccess | recoverCompositionHealth}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio API Gateway/BFF Composition Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Gateway composition workspace UX #oshun-web Platform on-call
Route contracts and composition APIs #oshun-bff BFF lead
Dependency health and fallback runtime #oshun-platform-sre SRE lead
Policy-tier/CAB governance controls #oshun-security Security + Compliance
Composition analytics and audit evidence #oshun-data-platform Data platform lead

7. Acceptance Criteria (OST-00272)#

OST-00272 is complete only when:

  1. Ownership and decision authority are documented.
  2. API gateway/BFF composition SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.