Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + Platform SRE + OSHUN
Security + Program Operations
Last Updated: 2026-02-23
Related:
docs/adr/ADR-0038-oshun-studio-background-jobs-and-progress-ux.mddocs/releases/v1/specs/studio-background-jobs-and-progress-ux-dependency-and-data-contract-map.mddocs/releases/v1/specs/studio-background-jobs-and-progress-ux-user-journeys-and-edge-cases.mdapps/oshun/web/src/components/studio/StudioBackgroundJobsProgressUxWorkspace.tsxapps/oshun/web/src/analytics/studioBackgroundJobsProgressUxTelemetry.ts
Purpose#
Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio background jobs and progress UX workflows.
This document is the execution artifact for OST-00208.
1. Ownership Model#
| Surface | Primary Owner | Backup Owner | Dependencies |
|---|---|---|---|
| Background-jobs workspace route | OSHUN Web | Platform Web | route contracts + progress/control UX orchestration |
| Job admission and queue orchestration | OSHUN BFF | Platform SRE | queue APIs + workflow identity contracts |
| Progress and heartbeat aggregation | Platform SRE | BFF | worker heartbeats + progress stream adapters |
| Control-action policy enforcement | Security + Compliance | Program Operations | RBAC/policy bundles + entitlement checks |
| Escalation routing and notifications | Program Operations + SRE | Security | notification center + on-call routing contracts |
| Governance evidence export pipeline | Data Platform + Security | Platform SRE | telemetry ingestion + retention/export controls |
2. Core SLOs#
| Metric | Target | Alert Threshold |
|---|---|---|
| Background-job flow success rate | >= 99.0% | < 98.0% over 15 min |
| Lifecycle transition reliability | >= 99.5% | < 99.0% over 15 min |
| Background-job flow p95 latency | <= 2400ms | > 3400ms over 15 min |
| Heartbeat freshness integrity | >= 99.5% | < 99.0% over 15 min |
| Control-action idempotency integrity | >= 99.5% | < 99.0% over 15 min |
| Escalation and governance evidence coverage | >= 99.5% | < 99.0% over 15 min |
3. Operational Playbooks#
3.1 Background Job Submission Failures#
Trigger: repeated BACKGROUND_JOB_SUBMISSION_FAILED outcomes.
Actions:
- Validate envelope identity (
jobId,workflowId,correlationId) and routing metadata. - Inspect queue-admission API logs for schema or policy rejection drift.
- Re-run submission in preview channel with diagnostics enabled.
- Block production promotion until submission integrity stabilizes.
3.2 Progress Heartbeat Degradation#
Trigger: repeated BACKGROUND_JOB_HEARTBEAT_STREAM_FAILED outcomes.
Actions:
- Validate worker heartbeat emitters and progress stream backpressure.
- Inspect freshness windows for stale heartbeat threshold drift.
- Re-run heartbeat stream with worker tracing enabled.
- Trigger stuck-job escalation if freshness SLO remains breached.
3.3 Control-Action Failures#
Trigger: repeated BACKGROUND_JOB_CONTROL_ACTION_FAILED outcomes.
Actions:
- Validate control action payload, policy mapping, and idempotency keys.
- Inspect control-plane state transitions for duplicate/reordered updates.
- Re-run control action with constrained blast radius.
- Disable risky action types (
cancel,reassign) until stability recovers.
3.4 Recovery Workflow Failures#
Trigger: repeated BACKGROUND_JOB_RECOVERY_FAILED outcomes.
Actions:
- Validate blocked/failed state prerequisites and lineage continuity checks.
- Inspect retry/resume logic for identity or correlation drift.
- Re-run recovery with reliability reviewer oversight.
- Escalate to Platform SRE if recovery loops persist.
3.5 Escalation/Governance Closure Failures#
Trigger: repeated BACKGROUND_JOB_GOVERNANCE_CLOSURE_FAILED outcomes.
Actions:
- Validate escalation severity, routing target, and acknowledgement fields.
- Inspect governance package for missing control-action or lifecycle evidence.
- Re-run closure flow with compliance auditor oversight.
- Block production closure exports until evidence validation passes.
3.6 Policy Denial Spike#
Trigger: unexpected rise in ROLE_POLICY_DENIED,
CRITICAL_PRIORITY_SUBMISSION_POLICY_DENIED, PRODUCTION_CANCEL_POLICY_DENIED,
PRODUCTION_CRITICAL_RECOVERY_POLICY_DENIED, or
PRODUCTION_GOVERNANCE_CLOSURE_POLICY_DENIED.
Actions:
- Compare flow allowlists against active RBAC and policy bundles.
- Inspect recent policy release changes for drift.
- Apply policy hotfix or roll back drifted policy bundle.
- Confirm deny-rate normalization and impacted flow recovery.
3.7 Telemetry and Audit Degradation#
Trigger: missing or delayed background-job telemetry/audit events.
Actions:
- Validate analytics sink health and ingestion backlog depth.
- Enable degraded-observability warning in background-jobs workspace.
- Replay buffered events where supported.
- Confirm SLO and evidence stream completeness after recovery.
4. Release Checklist#
- Validate
/studio/background-jobs-progress-uxrenders and executes required flows. - Validate optimistic, error, and retry behavior for each flow.
- Validate critical-priority, production control, and governance policy enforcement.
- Validate telemetry/audit emissions and SLO panel updates.
- Validate integration/e2e/accessibility evidence artifacts.
5. Incident Communication Templates#
Incident Start#
[P{severity}] Studio Background Jobs Incident
Detected: {timestamp}
Impact: {job-submit | progress-heartbeat | control-action | recovery | governance-closure}
Suspected cause: {policy drift | queue regression | heartbeat degradation | escalation routing failure}
Owner: {on-call}
Incident Update#
[P{severity}] Studio Background Jobs Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {submitBackgroundJob | streamProgressHeartbeat | executeJobControlAction | recoverBlockedOrFailedJob | escalateAndCloseGovernanceEvidence}
ETA: {time}
Incident Resolved#
[P{severity}] Studio Background Jobs Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}
6. Escalation Paths#
| Area | Primary Channel | Escalate To |
|---|---|---|
| Web runtime and workspace UX | #oshun-web |
Platform on-call |
| Job admission/queue orchestration | #oshun-bff |
BFF lead |
| Progress heartbeat and worker health | #oshun-platform-sre |
SRE lead |
| Policy/entitlement governance | #oshun-security |
Security lead |
| Escalation and evidence pipelines | #oshun-data-platform |
Data platform/SRE lead |
7. Acceptance Criteria (OST-00208)#
OST-00208 is complete only when:
- Ownership and decision authority are documented.
- Background-job/progress SLOs and alert thresholds are documented.
- At least five playbooks are documented for major failure modes.
- Release checklist, communication templates, and escalation paths exist.