# Runbook — Shell Outage

> Customer or admin shell is unreachable, unable to bootstrap, or returning 5xx
> on critical routes. Affects `shell_web`, `shell_pwa`, `shell_mobile`,
> `shell_admin_web`, or `shell_admin_mobile`.

## Scope

Use this runbook when the shell itself — not a single domain or assistant — is
the failure boundary. Examples:

- HTML/CSS/JS bundle 5xx from the CDN or origin.
- BFF bootstrap (`/api/bff/bootstrap`) returns 5xx, times out, or returns
  malformed JSON.
- Service worker fails to register or returns stale assets in a release window.
- Auth session cookie cannot be issued or refreshed.
- Mobile shell cannot reach update manifest and falls below the minimum
  supported build.

If only a single domain card (Tara/Veritas/etc.) is broken, use the matching
domain runbook instead. If the assistant is the failure boundary, use
`assistant-failure.md`.

## Severity classification

| Sev  | Condition                                                                                                     | Targets                            |
| ---- | ------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| sev1 | >5% of customer sessions cannot bootstrap for >5 min, OR admin shell unreachable during an active incident.   | shell\_\*, bff_consumer, bff_admin |
| sev2 | Bootstrap success ratio 95–99% for >10 min, OR PWA stuck on stale bundle for >15 min.                         | shell\_\*, cdn, bff_consumer       |
| sev3 | Single platform regression (e.g., iOS-only) below 99% bootstrap, OR home rail render-error rate above budget. | shell_mobile, shell_pwa            |

The ratio is
`count(shell.bootstrap.completed{success=true}) / count(shell.bootstrap.started)`
per the customer taxonomy.

## Detection signals

- Alerts: any `service.shell_*` health-target alert from
  `libs/oshun/analytics/src/dashboards-service-health.ts`; the
  `shell.bootstrap.failed` rate panel on the customer-KPI dashboard.
- Synthetic probes: `probe.shell_web.bootstrap` and
  `probe.shell_mobile.bootstrap` in the synthetic-monitor suite.
- Customer support inbox: spike on tickets tagged
  `support.case.tags.app_wont_load` or `support.case.tags.blank_screen`.
- CDN: origin error ratio > 2% on `shell_web` or asset 5xx > 1% over 5 min.

## Initial triage (first 5 minutes)

1. Open the on-call rotation for `oshun-shell-web-oncall` (or the matching
   shell-target on-call from `incident-ownership-manifest.ts`) and confirm
   primary acknowledged.
2. Open `dashboards-service-health.ts` shell panel — record the failing target,
   p99 latency, error ratio, and the active build/version per platform.
3. Decide blast radius:
   - **One platform** (e.g., only iOS) → suspect a platform release.
   - **All platforms, one region** → suspect CDN/edge or regional dependency.
   - **All platforms, all regions** → suspect BFF bootstrap or shared dependency
     (auth, config service, Postgres primary).
4. Page `bff_consumer` or `bff_admin` on-call (per
   `incident-ownership-manifest.ts`) if the BFF is implicated.

## Diagnosis

1. Confirm the active release:
   - Customer web: read `X-Oshun-Release` header from the CDN.
   - Mobile: check the active minimum-version row in the release-taxonomy
     manifest.
2. Check recent deploys (last 60 min) — release-taxonomy emits
   `release.publish.completed` and `release.canary.advanced`. Correlate against
   the bootstrap-success drop.
3. Inspect BFF bootstrap traces (OpenTelemetry span `bff.consumer.bootstrap`) —
   look for the slowest downstream dependency span (Postgres, Redis,
   feature-flag, auth, evidence index).
4. Inspect CDN logs for origin error ratio per edge POP. If only one POP is bad,
   instruct CDN failover.
5. Inspect service-worker telemetry (`pwa.sw.activated`,
   `pwa.sw.activation_failed`) for stale or broken activation.
6. Read the most recent change-management entries in `release-taxonomy.ts` —
   kill-switch state, config flags toggled, and tenant rollout cohorts. A
   misconfigured flag is the most common shell outage cause.

## Mitigation

1. **Rollback** (default mitigation). Use the matching release-taxonomy rollback
   hook for the affected target: web canary → previous stable build; mobile →
   revert minimum-version row; PWA → publish service-worker `cleanup` build that
   purges stale caches.
2. **Kill-switch a recent flag**: if a feature flag in `release-taxonomy.ts` was
   advanced in the last hour, set its cohort to 0% via the flag-admin API and
   observe bootstrap recovery. Document the flag ID and operator in the incident
   log.
3. **CDN edge failover**: instruct the CDN to fail over the affected POPs to a
   secondary; do not modify origins.
4. **Status banner**: publish the `incident.customer.status_page.investigating`
   template and the `incident.customer.in_app.investigating` in-app banner with
   the canonical `bodyPlaceholders` filled in (service, impact_summary,
   next_update_at). Banner copy is owned by the reliability lead; do not
   improvise.
5. **Mobile minimum-version path**: if the rollback also raises the minimum
   supported build, schedule the forced-update banner via the release-taxonomy
   minimum-version event and verify the in-app forced-update path renders.
6. **Auth-session preservation**: never invalidate sessions during a shell
   outage. If a config push must clear sessions, capture the customer-impact
   estimate first and reclassify to a sev1.

## Communication cadence

| Phase         | Template ID                                                  | Audience    | Cadence          |
| ------------- | ------------------------------------------------------------ | ----------- | ---------------- |
| Investigating | `incident.customer.status_page.investigating`                | status_page | within 5 min     |
| Investigating | `incident.customer.in_app.investigating`                     | customer    | within 5 min     |
| Internal      | `incident.internal.slack.investigating`                      | internal    | within 2 min     |
| Identified    | `incident.customer.status_page.identified`                   | status_page | within 30 min    |
| Mitigating    | `incident.customer.status_page.mitigating`                   | status_page | every 30 min     |
| Monitoring    | `incident.customer.status_page.monitoring`                   | status_page | when mitigated   |
| Resolved      | `incident.customer.status_page.resolved` + `.email.resolved` | both        | within 60 min    |
| Postmortem    | `incident.customer.status_page.postmortem`                   | status_page | within 5 biz day |

Use the canonical template IDs from
`libs/oshun/analytics/src/incident-ownership-manifest.ts` — do not invent ad-hoc
copy.

## Escalation

Standard escalation from `incident-ownership-manifest.ts` applies:

1. Primary on-call (5 min)
2. Secondary on-call (5 min)
3. Engineering manager (15 min)
4. Incident commander (30 min)
5. Executive on-call (60 min)

A sev1 declared at minute 0 reaches incident commander by minute 25.

## Recovery verification

The incident is `monitoring` once:

- Bootstrap success ratio is ≥ 99.5% across all platforms for 15 min.
- CDN origin error ratio is < 0.5% for 15 min.
- No new customer support tickets tagged `app_wont_load` in the last 10 min.
- Synthetic probes `probe.shell_*.bootstrap` green across all regions.

It is `resolved` once the above hold for 60 min and the canary advances back to
baseline.

## Post-incident

- File postmortem in `docs/incidents/` within 5 business days; link the incident
  ID, all release-taxonomy and flag-admin actions taken, the affected cohort
  estimate, and remediation owners.
- Attach the audit export of all `release.*` and `release-taxonomy.*` events in
  the incident window.
- Update `dashboards-service-health.ts` if a new signal is needed.
- Convert any improvised commands into release-taxonomy rollback hooks.
- If a feature flag was implicated, raise a rollout-policy ticket (cohort size,
  ramp speed, kill-switch criteria).