Runbook — Shell Outage

Customer or admin shell is unreachable, unable to bootstrap, or returning 5xx on critical routes. Affects shell_web, shell_pwa, shell_mobile, shell_admin_web, or shell_admin_mobile.

Scope#

Use this runbook when the shell itself — not a single domain or assistant — is the failure boundary. Examples:

• HTML/CSS/JS bundle 5xx from the CDN or origin.
• BFF bootstrap (/api/bff/bootstrap) returns 5xx, times out, or returns malformed JSON.
• Service worker fails to register or returns stale assets in a release window.
• Auth session cookie cannot be issued or refreshed.
• Mobile shell cannot reach update manifest and falls below the minimum supported build.

If only a single domain card (Tara/Veritas/etc.) is broken, use the matching domain runbook instead. If the assistant is the failure boundary, use assistant-failure.md.

Severity classification#

The ratio is count(shell.bootstrap.completed{success=true}) / count(shell.bootstrap.started) per the customer taxonomy.

Detection signals#

• Alerts: any service.shell_* health-target alert from libs/oshun/analytics/src/dashboards-service-health.ts; the shell.bootstrap.failed rate panel on the customer-KPI dashboard.
• Synthetic probes: probe.shell_web.bootstrap and probe.shell_mobile.bootstrap in the synthetic-monitor suite.
• Customer support inbox: spike on tickets tagged support.case.tags.app_wont_load or support.case.tags.blank_screen.
• CDN: origin error ratio > 2% on shell_web or asset 5xx > 1% over 5 min.

Initial triage (first 5 minutes)#

1. Open the on-call rotation for oshun-shell-web-oncall (or the matching shell-target on-call from incident-ownership-manifest.ts) and confirm primary acknowledged.
2. Open dashboards-service-health.ts shell panel — record the failing target, p99 latency, error ratio, and the active build/version per platform.
3. Decide blast radius:
   • One platform (e.g., only iOS) → suspect a platform release.
   • All platforms, one region → suspect CDN/edge or regional dependency.
   • All platforms, all regions → suspect BFF bootstrap or shared dependency (auth, config service, Postgres primary).
4. Page bff_consumer or bff_admin on-call (per incident-ownership-manifest.ts) if the BFF is implicated.

Diagnosis#

1. Confirm the active release:
   • Customer web: read X-Oshun-Release header from the CDN.
   • Mobile: check the active minimum-version row in the release-taxonomy manifest.
2. Check recent deploys (last 60 min) — release-taxonomy emits release.publish.completed and release.canary.advanced. Correlate against the bootstrap-success drop.
3. Inspect BFF bootstrap traces (OpenTelemetry span bff.consumer.bootstrap) — look for the slowest downstream dependency span (Postgres, Redis, feature-flag, auth, evidence index).
4. Inspect CDN logs for origin error ratio per edge POP. If only one POP is bad, instruct CDN failover.
5. Inspect service-worker telemetry (pwa.sw.activated, pwa.sw.activation_failed) for stale or broken activation.
6. Read the most recent change-management entries in release-taxonomy.ts — kill-switch state, config flags toggled, and tenant rollout cohorts. A misconfigured flag is the most common shell outage cause.

Mitigation#

1. Rollback (default mitigation). Use the matching release-taxonomy rollback hook for the affected target: web canary → previous stable build; mobile → revert minimum-version row; PWA → publish service-worker cleanup build that purges stale caches.
2. Kill-switch a recent flag: if a feature flag in release-taxonomy.ts was advanced in the last hour, set its cohort to 0% via the flag-admin API and observe bootstrap recovery. Document the flag ID and operator in the incident log.
3. CDN edge failover: instruct the CDN to fail over the affected POPs to a secondary; do not modify origins.
4. Status banner: publish the incident.customer.status_page.investigating template and the incident.customer.in_app.investigating in-app banner with the canonical bodyPlaceholders filled in (service, impact_summary, next_update_at). Banner copy is owned by the reliability lead; do not improvise.
5. Mobile minimum-version path: if the rollback also raises the minimum supported build, schedule the forced-update banner via the release-taxonomy minimum-version event and verify the in-app forced-update path renders.
6. Auth-session preservation: never invalidate sessions during a shell outage. If a config push must clear sessions, capture the customer-impact estimate first and reclassify to a sev1.

Communication cadence#

Use the canonical template IDs from libs/oshun/analytics/src/incident-ownership-manifest.ts — do not invent ad-hoc copy.

Escalation#

Standard escalation from incident-ownership-manifest.ts applies:

1. Primary on-call (5 min)
2. Secondary on-call (5 min)
3. Engineering manager (15 min)
4. Incident commander (30 min)
5. Executive on-call (60 min)

A sev1 declared at minute 0 reaches incident commander by minute 25.

Recovery verification#

The incident is monitoring once:

• Bootstrap success ratio is ≥ 99.5% across all platforms for 15 min.
• CDN origin error ratio is < 0.5% for 15 min.
• No new customer support tickets tagged app_wont_load in the last 10 min.
• Synthetic probes probe.shell_*.bootstrap green across all regions.

It is resolved once the above hold for 60 min and the canary advances back to baseline.

Post-incident#

• File postmortem in docs/incidents/ within 5 business days; link the incident ID, all release-taxonomy and flag-admin actions taken, the affected cohort estimate, and remediation owners.
• Attach the audit export of all release.* and release-taxonomy.* events in the incident window.
• Update dashboards-service-health.ts if a new signal is needed.
• Convert any improvised commands into release-taxonomy rollback hooks.
• If a feature flag was implicated, raise a rollout-policy ticket (cohort size, ramp speed, kill-switch criteria).