Runbook — Grounding Failure

Sophia grounded-answer pipeline produces low-confidence, missing-citation, stale-evidence, or hallucinated output above SLO. Use when grounding — not the upstream assistant model — is the failure boundary.

Scope#

This runbook covers degradation of the grounded-answer path that powers Veritas evidence rails, Tara reflection citations, Nisaba passage references, Metis tutor source binding, and the assistant grounded mode. Failure modes:

• Citation flag rate spike (a returned citation does not match the bound source).
• Grounding confidence below the min_confidence floor for >5% of grounded responses.
• Stale-evidence index (vector index lag > 15 min behind the source-of-truth store).
• Sophia adapter timeout, rate limit, or 5xx > 1%.
• Retracted/corrected source still served in a published answer.

If the assistant model itself is failing, use assistant-failure.md. If a specific provider is the failure boundary, use provider-failover.md.

Severity classification#

Thresholds come from the evaluation-manifest.ts benchmark scope grounded_answers (exact-match 0.7+, stance accuracy 0.85+) and from the grounding-low-confidence-spike and citation-flag-rate alerts in alerts-manifest.ts.

Detection signals#

• Alerts: grounding-low-confidence-spike, citation-flag-rate, grounding-fallback from libs/oshun/analytics/src/alerts-manifest.ts.
• Dashboards: dashboards-service-health.ts Sophia panel; the assistant evidence-quality panel from dashboards-assistant.ts.
• Customer signals: evidence.confidence.below_threshold and evidence.citation.mismatch_reported events from customer-taxonomy.ts.
• Admin signals: a moderation reviewer flags review.evidence.disputed.

Initial triage (first 5 minutes)#

1. Acknowledge oshun-sophia-oncall.
2. Determine failure boundary:
   • Confidence drop alone → upstream model or retrieval drift.
   • Citation mismatch alone → index or chunker drift.
   • Both → end-to-end pipeline outage (escalate immediately).
3. Read the active Sophia release row in release-taxonomy.ts. If a model or retriever was promoted in the last 60 min, that is the prime suspect.
4. If retracted-source-served is suspected, immediately disable grounded publication (mitigation step 1) before investigating further.

Diagnosis#

1. Pull the last hour of evidence.citation.mismatch_reported events, grouped by source ID. A clustered set of mismatches against the same source bundle implicates an index/chunker issue against that bundle.
2. Inspect the Sophia retriever trace (sophia.retriever.query/sophia.retriever.rerank) for p95 latency and the per-stage hit rate. A collapsed rerank score distribution implicates the reranker.
3. Compare current evaluation-manifest.ts grounded_answers benchmark scores against the last release row in release-taxonomy.ts. A score drop below the regression threshold blocks the active version and reverts to the previous certified retriever.
4. Compare vector-index lag (dependency_search_index health target) against the source-of-truth store. >15 min lag explains stale evidence.
5. Pull a sample of 20 grounded responses where confidence dropped. Inspect the bound chunks against the source-of-truth excerpts. If the chunks no longer match the published source text, the chunker or storage has drifted.

Mitigation#

1. Stop the bleed: disable grounded publication via the Sophia publication_gate kill switch. Customer surfaces fall back to the source-inspection-only view; assistant grounded mode degrades to non-streaming text answers with explicit "evidence unavailable" disclosure from customer-taxonomy.ts event evidence.unavailable_shown.
2. Revert the model/retriever: if release-taxonomy implicates a recent promotion, run the rollback hook for that target. The model-rollback alert runbook is the canonical hook; this runbook delegates to it for the actual rollback command.
3. Force a re-index for the affected bundle: enqueue sophia.reindex.requested for the source IDs implicated by the mismatch cluster; observe sophia.reindex.completed and the recovery of citation match rate.
4. Suppress retracted sources: if the cause is a retraction cascade, verify the retraction event was published to the evidence store. The retraction-cascade validator in Sophia rejects republication of retracted source IDs.
5. Re-grounding queue: enqueue customer-visible answers from the impact window for re-grounding. The publisher pauses customer-visible republish until each answer is recertified or marked evidence_unavailable.

Communication cadence#

If a retracted source was served, the customer-email resolved template must identify the affected content scope (no individual user data) and link the Sophia provenance bundle for the corrected answer.

Escalation#

Standard escalation from incident-ownership-manifest.ts. For sev1 (retracted source served, or confidence floor breached for >5 min), pre-emptively page the trust-and-safety lead in parallel with reliability — the customer-visible correction sits with T&S.

Recovery verification#

• Citation flag rate < 0.5% for 15 min.
• Grounding fallback rate < 1% for 15 min.
• grounded_answers benchmark passes (evaluation-manifest.ts thresholds).
• All re-grounded customer-visible answers either recertified or marked evidence-unavailable.
• No new evidence.citation.mismatch_reported clusters for 30 min.

Post-incident#

• Postmortem must include: regression sample IDs (anonymized), retraction cascade audit, re-grounding queue length and time-to-clear, the model or retriever version that regressed, and the recertified version.
• Update evaluation-manifest.ts thresholds if the incident revealed a gap in the benchmark coverage.
• If the cause was a missed retraction, add the source to the retraction-set fixture in the evaluation pipeline so future promotions catch it.