# OSHUN Studio Background Jobs and Progress UX Runbook

Status: Active  
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN
Security + Program Operations  
Last Updated: 2026-02-23  
Related:

- `docs/adr/ADR-0038-oshun-studio-background-jobs-and-progress-ux.md`
- `docs/releases/v1/specs/studio-background-jobs-and-progress-ux-dependency-and-data-contract-map.md`
- `docs/releases/v1/specs/studio-background-jobs-and-progress-ux-user-journeys-and-edge-cases.md`
- `apps/oshun/web/src/components/studio/StudioBackgroundJobsProgressUxWorkspace.tsx`
- `apps/oshun/web/src/analytics/studioBackgroundJobsProgressUxTelemetry.ts`

## Purpose

Define operational ownership, SLOs, incident playbooks, and escalation paths for
Studio background jobs and progress UX workflows.

This document is the execution artifact for `OST-00208`.

## 1. Ownership Model

| Surface                               | Primary Owner            | Backup Owner       | Dependencies                                        |
| ------------------------------------- | ------------------------ | ------------------ | --------------------------------------------------- |
| Background-jobs workspace route       | OSHUN Web                | Platform Web       | route contracts + progress/control UX orchestration |
| Job admission and queue orchestration | OSHUN BFF                | Platform SRE       | queue APIs + workflow identity contracts            |
| Progress and heartbeat aggregation    | Platform SRE             | BFF                | worker heartbeats + progress stream adapters        |
| Control-action policy enforcement     | Security + Compliance    | Program Operations | RBAC/policy bundles + entitlement checks            |
| Escalation routing and notifications  | Program Operations + SRE | Security           | notification center + on-call routing contracts     |
| Governance evidence export pipeline   | Data Platform + Security | Platform SRE       | telemetry ingestion + retention/export controls     |

## 2. Core SLOs

| Metric                                      | Target    | Alert Threshold      |
| ------------------------------------------- | --------- | -------------------- |
| Background-job flow success rate            | >= 99.0%  | < 98.0% over 15 min  |
| Lifecycle transition reliability            | >= 99.5%  | < 99.0% over 15 min  |
| Background-job flow p95 latency             | <= 2400ms | > 3400ms over 15 min |
| Heartbeat freshness integrity               | >= 99.5%  | < 99.0% over 15 min  |
| Control-action idempotency integrity        | >= 99.5%  | < 99.0% over 15 min  |
| Escalation and governance evidence coverage | >= 99.5%  | < 99.0% over 15 min  |

## 3. Operational Playbooks

### 3.1 Background Job Submission Failures

Trigger: repeated `BACKGROUND_JOB_SUBMISSION_FAILED` outcomes.

Actions:

1. Validate envelope identity (`jobId`, `workflowId`, `correlationId`) and
   routing metadata.
2. Inspect queue-admission API logs for schema or policy rejection drift.
3. Re-run submission in preview channel with diagnostics enabled.
4. Block production promotion until submission integrity stabilizes.

### 3.2 Progress Heartbeat Degradation

Trigger: repeated `BACKGROUND_JOB_HEARTBEAT_STREAM_FAILED` outcomes.

Actions:

1. Validate worker heartbeat emitters and progress stream backpressure.
2. Inspect freshness windows for stale heartbeat threshold drift.
3. Re-run heartbeat stream with worker tracing enabled.
4. Trigger stuck-job escalation if freshness SLO remains breached.

### 3.3 Control-Action Failures

Trigger: repeated `BACKGROUND_JOB_CONTROL_ACTION_FAILED` outcomes.

Actions:

1. Validate control action payload, policy mapping, and idempotency keys.
2. Inspect control-plane state transitions for duplicate/reordered updates.
3. Re-run control action with constrained blast radius.
4. Disable risky action types (`cancel`, `reassign`) until stability recovers.

### 3.4 Recovery Workflow Failures

Trigger: repeated `BACKGROUND_JOB_RECOVERY_FAILED` outcomes.

Actions:

1. Validate blocked/failed state prerequisites and lineage continuity checks.
2. Inspect retry/resume logic for identity or correlation drift.
3. Re-run recovery with reliability reviewer oversight.
4. Escalate to Platform SRE if recovery loops persist.

### 3.5 Escalation/Governance Closure Failures

Trigger: repeated `BACKGROUND_JOB_GOVERNANCE_CLOSURE_FAILED` outcomes.

Actions:

1. Validate escalation severity, routing target, and acknowledgement fields.
2. Inspect governance package for missing control-action or lifecycle evidence.
3. Re-run closure flow with compliance auditor oversight.
4. Block production closure exports until evidence validation passes.

### 3.6 Policy Denial Spike

Trigger: unexpected rise in `ROLE_POLICY_DENIED`,
`CRITICAL_PRIORITY_SUBMISSION_POLICY_DENIED`, `PRODUCTION_CANCEL_POLICY_DENIED`,
`PRODUCTION_CRITICAL_RECOVERY_POLICY_DENIED`, or
`PRODUCTION_GOVERNANCE_CLOSURE_POLICY_DENIED`.

Actions:

1. Compare flow allowlists against active RBAC and policy bundles.
2. Inspect recent policy release changes for drift.
3. Apply policy hotfix or roll back drifted policy bundle.
4. Confirm deny-rate normalization and impacted flow recovery.

### 3.7 Telemetry and Audit Degradation

Trigger: missing or delayed background-job telemetry/audit events.

Actions:

1. Validate analytics sink health and ingestion backlog depth.
2. Enable degraded-observability warning in background-jobs workspace.
3. Replay buffered events where supported.
4. Confirm SLO and evidence stream completeness after recovery.

## 4. Release Checklist

- [ ] Validate `/studio/background-jobs-progress-ux` renders and executes
      required flows.
- [ ] Validate optimistic, error, and retry behavior for each flow.
- [ ] Validate critical-priority, production control, and governance policy
      enforcement.
- [ ] Validate telemetry/audit emissions and SLO panel updates.
- [ ] Validate integration/e2e/accessibility evidence artifacts.

## 5. Incident Communication Templates

### Incident Start

```
[P{severity}] Studio Background Jobs Incident
Detected: {timestamp}
Impact: {job-submit | progress-heartbeat | control-action | recovery | governance-closure}
Suspected cause: {policy drift | queue regression | heartbeat degradation | escalation routing failure}
Owner: {on-call}
```

### Incident Update

```
[P{severity}] Studio Background Jobs Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {submitBackgroundJob | streamProgressHeartbeat | executeJobControlAction | recoverBlockedOrFailedJob | escalateAndCloseGovernanceEvidence}
ETA: {time}
```

### Incident Resolved

```
[P{severity}] Studio Background Jobs Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
```

## 6. Escalation Paths

| Area                                 | Primary Channel        | Escalate To            |
| ------------------------------------ | ---------------------- | ---------------------- |
| Web runtime and workspace UX         | `#oshun-web`           | Platform on-call       |
| Job admission/queue orchestration    | `#oshun-bff`           | BFF lead               |
| Progress heartbeat and worker health | `#oshun-platform-sre`  | SRE lead               |
| Policy/entitlement governance        | `#oshun-security`      | Security lead          |
| Escalation and evidence pipelines    | `#oshun-data-platform` | Data platform/SRE lead |

## 7. Acceptance Criteria (OST-00208)

`OST-00208` is complete only when:

1. Ownership and decision authority are documented.
2. Background-job/progress SLOs and alert thresholds are documented.
3. At least five playbooks are documented for major failure modes.
4. Release checklist, communication templates, and escalation paths exist.
