Eve library

OSHUN Studio Background Jobs and Progress UX Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0038-oshun-studio-background-jobs-and-progress-ux.md
  • docs/releases/v1/specs/studio-background-jobs-and-progress-ux-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-background-jobs-and-progress-ux-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioBackgroundJobsProgressUxWorkspace.tsx
  • apps/oshun/web/src/analytics/studioBackgroundJobsProgressUxTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio background jobs and progress UX workflows.

This document is the execution artifact for OST-00208.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Background-jobs workspace route OSHUN Web Platform Web route contracts + progress/control UX orchestration
Job admission and queue orchestration OSHUN BFF Platform SRE queue APIs + workflow identity contracts
Progress and heartbeat aggregation Platform SRE BFF worker heartbeats + progress stream adapters
Control-action policy enforcement Security + Compliance Program Operations RBAC/policy bundles + entitlement checks
Escalation routing and notifications Program Operations + SRE Security notification center + on-call routing contracts
Governance evidence export pipeline Data Platform + Security Platform SRE telemetry ingestion + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
Background-job flow success rate >= 99.0% < 98.0% over 15 min
Lifecycle transition reliability >= 99.5% < 99.0% over 15 min
Background-job flow p95 latency <= 2400ms > 3400ms over 15 min
Heartbeat freshness integrity >= 99.5% < 99.0% over 15 min
Control-action idempotency integrity >= 99.5% < 99.0% over 15 min
Escalation and governance evidence coverage >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Background Job Submission Failures#

Trigger: repeated BACKGROUND_JOB_SUBMISSION_FAILED outcomes.

Actions:

  1. Validate envelope identity (jobId, workflowId, correlationId) and routing metadata.
  2. Inspect queue-admission API logs for schema or policy rejection drift.
  3. Re-run submission in preview channel with diagnostics enabled.
  4. Block production promotion until submission integrity stabilizes.

3.2 Progress Heartbeat Degradation#

Trigger: repeated BACKGROUND_JOB_HEARTBEAT_STREAM_FAILED outcomes.

Actions:

  1. Validate worker heartbeat emitters and progress stream backpressure.
  2. Inspect freshness windows for stale heartbeat threshold drift.
  3. Re-run heartbeat stream with worker tracing enabled.
  4. Trigger stuck-job escalation if freshness SLO remains breached.

3.3 Control-Action Failures#

Trigger: repeated BACKGROUND_JOB_CONTROL_ACTION_FAILED outcomes.

Actions:

  1. Validate control action payload, policy mapping, and idempotency keys.
  2. Inspect control-plane state transitions for duplicate/reordered updates.
  3. Re-run control action with constrained blast radius.
  4. Disable risky action types (cancel, reassign) until stability recovers.

3.4 Recovery Workflow Failures#

Trigger: repeated BACKGROUND_JOB_RECOVERY_FAILED outcomes.

Actions:

  1. Validate blocked/failed state prerequisites and lineage continuity checks.
  2. Inspect retry/resume logic for identity or correlation drift.
  3. Re-run recovery with reliability reviewer oversight.
  4. Escalate to Platform SRE if recovery loops persist.

3.5 Escalation/Governance Closure Failures#

Trigger: repeated BACKGROUND_JOB_GOVERNANCE_CLOSURE_FAILED outcomes.

Actions:

  1. Validate escalation severity, routing target, and acknowledgement fields.
  2. Inspect governance package for missing control-action or lifecycle evidence.
  3. Re-run closure flow with compliance auditor oversight.
  4. Block production closure exports until evidence validation passes.

3.6 Policy Denial Spike#

Trigger: unexpected rise in ROLE_POLICY_DENIED, CRITICAL_PRIORITY_SUBMISSION_POLICY_DENIED, PRODUCTION_CANCEL_POLICY_DENIED, PRODUCTION_CRITICAL_RECOVERY_POLICY_DENIED, or PRODUCTION_GOVERNANCE_CLOSURE_POLICY_DENIED.

Actions:

  1. Compare flow allowlists against active RBAC and policy bundles.
  2. Inspect recent policy release changes for drift.
  3. Apply policy hotfix or roll back drifted policy bundle.
  4. Confirm deny-rate normalization and impacted flow recovery.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed background-job telemetry/audit events.

Actions:

  1. Validate analytics sink health and ingestion backlog depth.
  2. Enable degraded-observability warning in background-jobs workspace.
  3. Replay buffered events where supported.
  4. Confirm SLO and evidence stream completeness after recovery.

4. Release Checklist#

  • Validate /studio/background-jobs-progress-ux renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate critical-priority, production control, and governance policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Background Jobs Incident
Detected: {timestamp}
Impact: {job-submit | progress-heartbeat | control-action | recovery | governance-closure}
Suspected cause: {policy drift | queue regression | heartbeat degradation | escalation routing failure}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Background Jobs Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {submitBackgroundJob | streamProgressHeartbeat | executeJobControlAction | recoverBlockedOrFailedJob | escalateAndCloseGovernanceEvidence}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Background Jobs Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and workspace UX #oshun-web Platform on-call
Job admission/queue orchestration #oshun-bff BFF lead
Progress heartbeat and worker health #oshun-platform-sre SRE lead
Policy/entitlement governance #oshun-security Security lead
Escalation and evidence pipelines #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00208)#

OST-00208 is complete only when:

  1. Ownership and decision authority are documented.
  2. Background-job/progress SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.