# Runbook — Assistant Failure

> Assistant streaming, persona, memory, or grounded-mode response path is
> failing for a material fraction of sessions. Use when the assistant itself —
> not the shell, not Sophia grounding — is the failure boundary.

## Scope

Covers assistant degradation across:

- Streaming token timeouts or repeated stalls (`assistant.stream.token`
  intertoken latency > SLO).
- Refusal-rate spike (policy or Lilith-persona rule mis-tuning).
- Persona drift (wrong tone, missing disclosure, identity confusion).
- Memory adapter (Iris) errors causing memory-off fallback.
- Grounded-mode unavailable for assistant sessions specifically (delegate to
  `grounding-failure.md` for the upstream cause but stay in this runbook for the
  assistant-surface mitigation).
- Voice/avatar mode failure where the assistant-product team owns the
  customer-visible mitigation.

For provider-side outages (network/auth/region), use `provider-failover.md`. For
shell-level breakage (bootstrap, route, auth), use `shell-outage.md`.

## Severity classification

| Sev  | Condition                                                                                                                                  |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| sev1 | >5% of assistant sessions fail to receive any tokens for >5 min, OR a persona violates AI-disclosure compliance in customer-visible flows. |
| sev1 | Memory adapter writes succeed against the wrong tenant, OR a refusal is bypassed for a Lilith-controlled policy class.                     |
| sev2 | Streaming stall rate above the SLO; refusal-rate change beyond ±2 sigma; voice/avatar fallback rate > 5%.                                  |
| sev3 | Latency-only regression with delivery success intact; cosmetic persona issues.                                                             |

Source events for these thresholds are `assistant.stream.token_received`,
`assistant.refusal.applied`, `assistant.persona.identity_confirmed`,
`assistant.memory.write_attempted` from `assistant-taxonomy.ts`.

## Detection signals

- Alerts: `assistant-provider-outage`, `persona-policy-violation` from
  `alerts-manifest.ts`.
- Dashboards: `dashboards-assistant.ts` quality + compliance panels — intertoken
  latency p90, turn success rate, disclosure compliance, refusal rate, memory
  write success.
- Customer signals: spike in `assistant.session.error_shown`,
  `assistant.disclosure.banner_dismissed`.
- Admin signals: T&S queue spike for `review.assistant.policy_appeal`.

## Initial triage (first 5 minutes)

1. Acknowledge `oshun-bff-assistant-oncall`.
2. Determine failure boundary:
   - Streaming stall → upstream provider, BFF, or token-distribution path.
   - Refusal spike → Lilith policy version or model alignment change.
   - Persona drift → persona-version regression (delegate to
     `persona-rollback.md` if identified).
   - Memory error → Iris adapter or downstream Postgres/Redis.
3. If a Lilith policy or persona promotion happened in the last 60 min, prepare
   the rollback command before further diagnosis.
4. If memory writes are crossing tenants, **stop all memory writes** before
   investigating (mitigation step 1).

## Diagnosis

1. Open the assistant trace for a failing session — span `bff.assistant.turn` →
   child spans for `lilith.policy.evaluate`, `iris.memory.fetch`,
   `sophia.retriever.query`, and the provider call. The slowest or failing child
   is the prime suspect.
2. Inspect the provider health from `release-taxonomy.ts` — provider state,
   region, current circuit breaker state, recent failover event.
3. Inspect the persona-rollout cohort: persona-taxonomy events
   `persona.version.activated` correlated with the failure-onset minute.
4. Inspect refusal classes: group `assistant.refusal.applied` by `policy_class`
   over the last 60 min; a single class spiking implicates a policy tightening.
5. Inspect memory writes: `iris.memory.write_attempted` grouped by tenant — any
   cross-tenant write triggers a sev1 reclassification.
6. Inspect voice/avatar paths if relevant: `assistant.voice.session_started` and
   `assistant.avatar.session_started` success rate; the voice/avatar teams own
   the deeper diagnostic.

## Mitigation

1. **Memory write quarantine** (if cross-tenant write suspected): set the Iris
   kill switch `memory.writes_enabled=false` immediately. All subsequent
   sessions fall back to non-persistent memory and the
   `assistant.memory.write_quarantined` event is emitted to admin audit.
2. **Persona rollback**: if persona drift is implicated, follow
   `persona-rollback.md`.
3. **Policy rollback**: if a Lilith policy version is implicated, revert via the
   Lilith release-taxonomy hook and observe refusal-rate recovery.
4. **Provider failover**: if the upstream provider is failing, follow
   `provider-failover.md` and verify the assistant cohort moves to the
   alternate.
5. **Grounded-mode disable**: if grounded mode is the failure surface, disable
   grounded-mode for assistant sessions via the assistant
   `grounded_mode_enabled` flag while keeping non-grounded answers available.
   Disclosure shifts to non-grounded copy automatically per
   `customer-taxonomy.ts`.
6. **Streaming fallback**: enable non-streaming responses
   (`assistant.stream_mode=batch`) when intertoken latency stalls cannot be
   resolved within 30 min; this restores delivery at the cost of perceived
   latency.
7. **Voice/avatar fallback**: degrade voice → text and avatar → non-animated
   portrait per the assistant-fallback policy. The
   `assistant.voice.fallback_shown` and `assistant.avatar.fallback_shown` events
   emit for analytics.

## Communication cadence

Identical structure to `shell-outage.md`. For sev1 with disclosure or memory
implications, the resolved customer email must clearly state the cohort, the
remediation, and the option to delete affected memory entries. Coordinate copy
with the privacy lead before sending.

## Escalation

Standard escalation. For sev1 with persona/disclosure/memory implications,
pre-emptively page T&S and privacy in parallel with assistant_product.

## Recovery verification

- Turn success rate ≥ 99% for 15 min.
- Intertoken latency p90 within SLO for 15 min.
- Refusal rate within ±1 sigma of baseline.
- Memory writes succeeding with tenant integrity verified by a fresh
  cross-tenant audit pass (no `memory.cross_tenant_detected` events for 30 min).
- Voice/avatar fallback rate < 1%.

## Post-incident

- Postmortem must include the persona-version, policy-version, model-version,
  and provider-route active at incident open, plus the recovered versions.
- Attach the audit export of `assistant.*`, `iris.*`, and `lilith.*` events in
  the incident window.
- If memory was affected, file a privacy-impact note even if no data was exposed
  — the customer-visible memory state must be reconciled.
- If a refusal class was tightened in error, the Lilith policy review packet
  must explain the false-positive sample and the recertified policy.