Eve library

OSHUN Studio File and Media Ingestion Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0036-oshun-studio-file-and-media-ingestion.md
  • docs/releases/v1/specs/studio-file-and-media-ingestion-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-file-and-media-ingestion-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioFileMediaIngestionWorkspace.tsx
  • apps/oshun/web/src/analytics/studioFileMediaIngestionTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio file/media ingestion workflows.

This document is the execution artifact for OST-00192.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
File/media ingestion workspace route OSHUN Web Platform Web route contracts + ingestion UX orchestration
Ingestion envelope persistence OSHUN BFF Platform SRE ingestion APIs + metadata contracts
Validation and classification service Security + Compliance Platform SRE validation engine + policy bundles
Processing/transformation lifecycle Platform SRE + BFF Data Platform processing workers + storage/transcode pipelines
Governance evidence export Data Platform + Security Program Operations audit streams + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
File/media ingestion flow success rate >= 99.0% < 98.0% over 15 min
Validation gate reliability >= 99.5% < 99.0% over 15 min
Ingestion flow p95 latency <= 2400ms > 3400ms over 15 min
Processing readiness completion >= 99.0% < 98.0% over 15 min
Recovery replay/idempotency integrity >= 99.5% < 99.0% over 15 min
Ingestion audit event completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Ingestion Envelope Submit Failures#

Trigger: repeated INGESTION_SUBMIT_FAILED outcomes.

Actions:

  1. Validate envelope identity, owner, classification, and checksum metadata.
  2. Inspect source adapter output for missing fields or malformed values.
  3. Re-run submit flow in preview with diagnostics enabled.
  4. Block rollout until envelope validation stability normalizes.

3.2 Validation Gate Failures#

Trigger: repeated INGESTION_VALIDATION_FAILED outcomes.

Actions:

  1. Validate type, size, checksum, and policy check configuration.
  2. Inspect validation pipeline traces for deterministic failure code mapping.
  3. Re-run validation gate with policy diagnostics enabled.
  4. Pause ingestion promotion until gate reliability recovers.

3.3 Processing Lifecycle Failures#

Trigger: repeated INGESTION_PROCESSING_FAILED outcomes.

Actions:

  1. Validate worker queue health and transformation pipeline dependencies.
  2. Inspect lifecycle transition logs for deterministic state progression.
  3. Re-run processing flow with controlled test payload.
  4. Escalate to Platform SRE if readiness conversion remains unstable.

3.4 Recovery Failures#

Trigger: repeated INGESTION_RECOVERY_FAILED outcomes.

Actions:

  1. Validate retry/replay/rollback action contracts and idempotency invariants.
  2. Inspect recovery action traces for policy and lineage drift.
  3. Re-run recovery with restricted blast-radius controls.
  4. Freeze manual recovery actions until integrity checks pass.

3.5 Governance Export Failures#

Trigger: repeated INGESTION_GOVERNANCE_EXPORT_FAILED outcomes.

Actions:

  1. Validate evidence package completeness and retention tagging.
  2. Inspect export pipeline for schema and storage contract mismatches.
  3. Re-run export with compliance reviewer oversight.
  4. Block production evidence exports until validation passes.

3.6 Policy Denial Spike#

Trigger: unexpected rise in ROLE_POLICY_DENIED, CONFIDENTIAL_INGESTION_POLICY_DENIED, MANUAL_OVERRIDE_POLICY_DENIED, or PRODUCTION_GOVERNANCE_EXPORT_POLICY_DENIED.

Actions:

  1. Compare flow allowlists against active RBAC and policy bundles.
  2. Inspect recent policy release changes for drift.
  3. Apply policy hotfix or roll back drifted policy bundle.
  4. Confirm deny-rate normalization and workflow impact resolution.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed ingestion telemetry/audit events.

Actions:

  1. Validate analytics sink health and ingestion backlog.
  2. Enable degraded-observability warning in ingestion workspace.
  3. Replay buffered events where supported.
  4. Confirm SLO and evidence stream completeness after recovery.

4. Release Checklist#

  • Validate /studio/file-media-ingestion renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate classification, recovery, and governance policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio File/Media Ingestion Incident
Detected: {timestamp}
Impact: {envelope-submit | validation-gate | processing-lifecycle | recovery-action | governance-export}
Suspected cause: {policy drift | pipeline regression | retention export failure}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio File/Media Ingestion Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {submitIngestionEnvelope | executeValidationGate | runProcessingLifecycle | recoverFailedIngestion | exportGovernanceEvidence}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio File/Media Ingestion Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and ingestion workspace #oshun-web Platform on-call
Ingestion persistence and lifecycle #oshun-bff BFF lead
Validation and policy governance #oshun-security Security lead
Processing/recovery infrastructure #oshun-platform-sre SRE lead
Audit/compliance evidence pipelines #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00192)#

OST-00192 is complete only when:

  1. Ownership and decision authority are documented.
  2. File/media ingestion SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.