Eve library

OSHUN Studio Asset Preview Pipeline Runbook

Runbook

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN BFF + OSHUN Security + Program Operations
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0037-oshun-studio-asset-preview-pipeline.md
  • docs/releases/v1/specs/studio-asset-preview-pipeline-dependency-and-data-contract-map.md
  • docs/releases/v1/specs/studio-asset-preview-pipeline-user-journeys-and-edge-cases.md
  • apps/oshun/web/src/components/studio/StudioAssetPreviewPipelineWorkspace.tsx
  • apps/oshun/web/src/analytics/studioAssetPreviewPipelineTelemetry.ts

Purpose#

Define operational ownership, SLOs, incident playbooks, and escalation paths for Studio asset preview pipeline workflows.

This document is the execution artifact for OST-00200.

1. Ownership Model#

Surface Primary Owner Backup Owner Dependencies
Asset preview workspace route OSHUN Web Platform Web route contracts + preview UX orchestration
Preview request envelope persistence OSHUN BFF Platform SRE preview APIs + revision metadata contracts
Render lifecycle orchestration Platform SRE Rendering Platform render workers + lifecycle state services
Fidelity fallback policy governance Security + Compliance Studio Architecture profile policy bundles + entitlement checks
Cache invalidation and refresh Platform SRE + CDN Ops BFF cache services + revision-key lineage
Governance evidence export pipeline Data Platform + Security Program Operations audit streams + retention/export controls

2. Core SLOs#

Metric Target Alert Threshold
Asset preview flow success rate >= 99.0% < 98.0% over 15 min
Render lifecycle reliability >= 99.5% < 99.0% over 15 min
Asset preview flow p95 latency <= 2300ms > 3300ms over 15 min
Fidelity fallback determinism >= 99.5% < 99.0% over 15 min
Cache invalidation/refresh correctness >= 99.5% < 99.0% over 15 min
Asset preview audit event completeness >= 99.5% < 99.0% over 15 min

3. Operational Playbooks#

3.1 Preview Request Failures#

Trigger: repeated PREVIEW_REQUEST_FAILED outcomes.

Actions:

  1. Validate preview request envelope (previewId, assetRef, profile, classification, revision) completeness.
  2. Inspect request adapter output for missing revision or asset identity data.
  3. Re-run preview request flow in preview channel with diagnostics enabled.
  4. Block release promotion until request envelope validation stabilizes.

3.2 Render Lifecycle Failures#

Trigger: repeated PREVIEW_RENDER_LIFECYCLE_FAILED outcomes.

Actions:

  1. Validate queued -> rendering -> ready transition contracts and guardrails.
  2. Inspect render-worker traces for non-deterministic lifecycle transitions.
  3. Re-run lifecycle orchestration with state tracing enabled.
  4. Freeze production render promotions until lifecycle reliability recovers.

3.3 Fidelity Fallback Failures#

Trigger: repeated PREVIEW_FIDELITY_FALLBACK_FAILED outcomes.

Actions:

  1. Validate fallback profile mapping and deterministic downgrade rules.
  2. Inspect fallback execution traces for profile lineage drift.
  3. Re-run fallback flow with policy diagnostics enabled.
  4. Disable profile downgrade rollout when fallback integrity cannot be proven.

3.4 Cache Invalidation and Refresh Failures#

Trigger: repeated PREVIEW_CACHE_INVALIDATION_FAILED outcomes.

Actions:

  1. Validate invalidation reason, revision key, and refresh lineage contracts.
  2. Inspect cache service traces for stale-key retention or refresh races.
  3. Re-run cache invalidation flow with constrained blast radius.
  4. Escalate to CDN ops if refresh misses remain above alert threshold.

3.5 Governance Export Failures#

Trigger: repeated PREVIEW_GOVERNANCE_EXPORT_FAILED outcomes.

Actions:

  1. Validate evidence package completeness for lifecycle, fallback, and cache lineage metadata.
  2. Inspect export pipeline for schema or retention-tagging mismatches.
  3. Re-run evidence export with compliance reviewer oversight.
  4. Block production evidence export until package validation passes.

3.6 Policy Denial Spike#

Trigger: unexpected rise in ROLE_POLICY_DENIED, CONFIDENTIAL_PREVIEW_POLICY_DENIED, PRODUCTION_HIGH_FIDELITY_POLICY_DENIED, or PRODUCTION_GOVERNANCE_EXPORT_POLICY_DENIED.

Actions:

  1. Compare flow allowlists against active RBAC and policy bundles.
  2. Inspect recent policy release changes for drift.
  3. Apply policy hotfix or roll back drifted policy bundle.
  4. Confirm deny-rate normalization and impacted flow recovery.

3.7 Telemetry and Audit Degradation#

Trigger: missing or delayed asset preview telemetry and audit events.

Actions:

  1. Validate analytics sink health and ingestion backlog depth.
  2. Enable degraded-observability warning in preview workspace.
  3. Replay buffered events where supported.
  4. Confirm SLO and evidence stream completeness after recovery.

4. Release Checklist#

  • Validate /studio/asset-preview-pipeline renders and executes required flows.
  • Validate optimistic, error, and retry behavior for each flow.
  • Validate profile, fallback, cache, and governance policy enforcement.
  • Validate telemetry/audit emissions and SLO panel updates.
  • Validate integration/e2e/accessibility evidence artifacts.

5. Incident Communication Templates#

Incident Start#

text
[P{severity}] Studio Asset Preview Pipeline Incident
Detected: {timestamp}
Impact: {preview-request | render-lifecycle | fidelity-fallback | cache-refresh | governance-export}
Suspected cause: {policy drift | render pipeline regression | cache invalidation failure}
Owner: {on-call}

Incident Update#

text
[P{severity}] Studio Asset Preview Pipeline Update
Time: {timestamp}
Mitigation: {rollback | hotfix | degraded mode}
Affected flows: {requestPreviewRender | orchestrateRenderLifecycle | executeFidelityFallback | invalidateAndRefreshCache | exportPreviewGovernanceEvidence}
ETA: {time}

Incident Resolved#

text
[P{severity}] Studio Asset Preview Pipeline Incident Resolved
Resolved: {timestamp}
Root cause: {description}
Fix: {release/hotfix id}
Validation: {tests + telemetry checks}
Postmortem: {link}

6. Escalation Paths#

Area Primary Channel Escalate To
Web runtime and preview workspace #oshun-web Platform on-call
Preview request and metadata services #oshun-bff BFF lead
Render lifecycle and cache services #oshun-platform-sre SRE lead
Policy/fallback governance #oshun-security Security lead
Audit/compliance evidence pipelines #oshun-data-platform Data platform/SRE lead

7. Acceptance Criteria (OST-00200)#

OST-00200 is complete only when:

  1. Ownership and decision authority are documented.
  2. Asset preview pipeline SLOs and alert thresholds are documented.
  3. At least five playbooks are documented for major failure modes.
  4. Release checklist, communication templates, and escalation paths exist.