Runbook — Assistant Failure

Assistant streaming, persona, memory, or grounded-mode response path is failing for a material fraction of sessions. Use when the assistant itself — not the shell, not Sophia grounding — is the failure boundary.

Scope#

Covers assistant degradation across:

• Streaming token timeouts or repeated stalls (assistant.stream.token intertoken latency > SLO).
• Refusal-rate spike (policy or Lilith-persona rule mis-tuning).
• Persona drift (wrong tone, missing disclosure, identity confusion).
• Memory adapter (Iris) errors causing memory-off fallback.
• Grounded-mode unavailable for assistant sessions specifically (delegate to grounding-failure.md for the upstream cause but stay in this runbook for the assistant-surface mitigation).
• Voice/avatar mode failure where the assistant-product team owns the customer-visible mitigation.

For provider-side outages (network/auth/region), use provider-failover.md. For shell-level breakage (bootstrap, route, auth), use shell-outage.md.

Severity classification#

Source events for these thresholds are assistant.stream.token_received, assistant.refusal.applied, assistant.persona.identity_confirmed, assistant.memory.write_attempted from assistant-taxonomy.ts.

Detection signals#

• Alerts: assistant-provider-outage, persona-policy-violation from alerts-manifest.ts.
• Dashboards: dashboards-assistant.ts quality + compliance panels — intertoken latency p90, turn success rate, disclosure compliance, refusal rate, memory write success.
• Customer signals: spike in assistant.session.error_shown, assistant.disclosure.banner_dismissed.
• Admin signals: T&S queue spike for review.assistant.policy_appeal.

Initial triage (first 5 minutes)#

1. Acknowledge oshun-bff-assistant-oncall.
2. Determine failure boundary:
   • Streaming stall → upstream provider, BFF, or token-distribution path.
   • Refusal spike → Lilith policy version or model alignment change.
   • Persona drift → persona-version regression (delegate to persona-rollback.md if identified).
   • Memory error → Iris adapter or downstream Postgres/Redis.
3. If a Lilith policy or persona promotion happened in the last 60 min, prepare the rollback command before further diagnosis.
4. If memory writes are crossing tenants, stop all memory writes before investigating (mitigation step 1).

Diagnosis#

1. Open the assistant trace for a failing session — span bff.assistant.turn → child spans for lilith.policy.evaluate, iris.memory.fetch, sophia.retriever.query, and the provider call. The slowest or failing child is the prime suspect.
2. Inspect the provider health from release-taxonomy.ts — provider state, region, current circuit breaker state, recent failover event.
3. Inspect the persona-rollout cohort: persona-taxonomy events persona.version.activated correlated with the failure-onset minute.
4. Inspect refusal classes: group assistant.refusal.applied by policy_class over the last 60 min; a single class spiking implicates a policy tightening.
5. Inspect memory writes: iris.memory.write_attempted grouped by tenant — any cross-tenant write triggers a sev1 reclassification.
6. Inspect voice/avatar paths if relevant: assistant.voice.session_started and assistant.avatar.session_started success rate; the voice/avatar teams own the deeper diagnostic.

Mitigation#

1. Memory write quarantine (if cross-tenant write suspected): set the Iris kill switch memory.writes_enabled=false immediately. All subsequent sessions fall back to non-persistent memory and the assistant.memory.write_quarantined event is emitted to admin audit.
2. Persona rollback: if persona drift is implicated, follow persona-rollback.md.
3. Policy rollback: if a Lilith policy version is implicated, revert via the Lilith release-taxonomy hook and observe refusal-rate recovery.
4. Provider failover: if the upstream provider is failing, follow provider-failover.md and verify the assistant cohort moves to the alternate.
5. Grounded-mode disable: if grounded mode is the failure surface, disable grounded-mode for assistant sessions via the assistant grounded_mode_enabled flag while keeping non-grounded answers available. Disclosure shifts to non-grounded copy automatically per customer-taxonomy.ts.
6. Streaming fallback: enable non-streaming responses (assistant.stream_mode=batch) when intertoken latency stalls cannot be resolved within 30 min; this restores delivery at the cost of perceived latency.
7. Voice/avatar fallback: degrade voice → text and avatar → non-animated portrait per the assistant-fallback policy. The assistant.voice.fallback_shown and assistant.avatar.fallback_shown events emit for analytics.

Communication cadence#

Identical structure to shell-outage.md. For sev1 with disclosure or memory implications, the resolved customer email must clearly state the cohort, the remediation, and the option to delete affected memory entries. Coordinate copy with the privacy lead before sending.

Escalation#

Standard escalation. For sev1 with persona/disclosure/memory implications, pre-emptively page T&S and privacy in parallel with assistant_product.

Recovery verification#

• Turn success rate ≥ 99% for 15 min.
• Intertoken latency p90 within SLO for 15 min.
• Refusal rate within ±1 sigma of baseline.
• Memory writes succeeding with tenant integrity verified by a fresh cross-tenant audit pass (no memory.cross_tenant_detected events for 30 min).
• Voice/avatar fallback rate < 1%.

Post-incident#

• Postmortem must include the persona-version, policy-version, model-version, and provider-route active at incident open, plus the recovered versions.
• Attach the audit export of assistant.*, iris.*, and lilith.* events in the incident window.
• If memory was affected, file a privacy-impact note even if no data was exposed — the customer-visible memory state must be reconciled.
• If a refusal class was tightened in error, the Lilith policy review packet must explain the false-positive sample and the recertified policy.