# Runbook: V1 Search Index Loss (`gd:search-index-loss`)

Owner: workbench-platform on-call (`rota:workbench-platform`). Last reviewed:
2026-09-14. Version: 2.

> A derived store, rebuildable from the relational one, which is exactly why its
> recovery time is long and its recovery point is inherited: nothing is lost
> that the source does not also lose, and the wait is the reprojection.

**Two facts decide this recovery, and both are absences.**

1. **The index is in no backup.** `infra/hetzner/backup/offbox-backup.sh`
   captures PostgreSQL dumps, the Redis RDB, the MinIO mirror, content-service
   runs, computer-use snapshots and configuration. The Qdrant volume is not
   among them, and that is correct — a derived store restored from a stale copy
   is worse than an empty one, because it is wrong instead of obviously missing.
2. **There is an executable reindex boundary.** The BFF image contains
   `dist/reproject-personalization-vector-index.js`, built from
   `reproject-personalization-vector-index.ts`; `@oshun/bff` also exposes it as
   `reproject:personalization-index`. BFF boot runs the same reconciliation
   before HTTP is constructed.

## 1. Detection and declaration

**Detection signal.** `slo:search-availability` and `slo:index-freshness` are
the registered objectives, and freshness is the one that moves first: an index
that has stopped accepting projections still answers queries, with answers that
get older. The two capabilities that stop outright are "search the catalogue"
and "filter a board by facet".

`dep:search-index` carries `fallback` among its registered mitigations —
`dep:primary-store-scan`, the search standby. **The fallback is what makes this
incident quiet.** With the scan serving, search still works and is slower, so
the estate degrades rather than stops, and the outage can run for a long time
before anybody declares it.

- Detection authority: **workbench-platform on-call**, contact
  `rota:workbench-platform`.
- Declaration authority: **production operations duty officer**, contact
  `rota:prod-ops`.
- **Gap, stated.** `docs/operations/on-call.md` describes a weekly
  primary/secondary rotation and names no person and no rotation config; none of
  the register's `rota:` identifiers appears in it or in
  `docs/operations/alert-routing.md`. The name above is registered; the route
  from it to whoever is carrying the pager is not.

## 2. Communications

| Audience  | Within | Content                                                                             |
| --------- | ------ | ----------------------------------------------------------------------------------- |
| Actors    | 15 min | that search results are incomplete or stale, and which surfaces fall back to a scan |
| Operators | 5 min  | that this runbook is running, and by whom                                           |

Both travel over `dep:notification-channel`, which this scenario leaves
standing.

**Tell actors the results are incomplete.** A search that returns fewer rows
than it should is the failure an actor cannot see: an empty result reads as "no
such item" and not as "the index is down".

## 3. Dependency assumptions

- `dep:primary-store` is healthy. It is the reprojection source, and the
  reference graph runs `dep:search-index` → `dep:primary-store`: projected
  documents carry the id of the row they were built from. An index rebuilt from
  a store that is itself impaired is a second incident wearing the first one's
  clothes.
- `dep:primary-store-scan` is healthy and carrying the search load. The scan
  rests on the same relational store, so this assumption is the one above
  restated — which is why S10.10 records that "search degrades to a scan" is a
  plan for a search-index outage and not for a database one.

## 4. Recovery infrastructure

**In place.** Qdrant is a private, health-gated service in
`infra/hetzner/docker-compose.yml`. The BFF is ordered after it and production
requires the endpoint, API key, collection, dimensions, and embedding version.
Recovery starts Qdrant with an empty volume and fills it from PostgreSQL.

## 5. Secrets

- `OSHUN_PERSONALIZATION_QDRANT_API_KEY`, dimensions, and embedding-version
  configuration from the recovered stack environment.

Recoverable out of band: the `configuration-and-secrets` state class is in the
off-box backup.

## 6. Restore order

One dependency, `dep:search-index`, and its single ordering constraint is
against the source rather than within the set: **the relational store must be
healthy and current before the reprojection starts.** That is the reference
direction — the referenced store first — and it is the reverse of the capture
order, in which the referring store is captured first so that the skew leaves
orphans instead of dangling references.

Reprojecting from a store that is still restoring produces an index that is
internally consistent and wrong, and nothing downstream will notice.

Once PostgreSQL and Qdrant are healthy, run:

```bash
docker compose -p oshun-production \
  --env-file /opt/oshun/production/.env \
  -f /opt/oshun/bundle/docker-compose.yml \
  run --rm bff node dist/reproject-personalization-vector-index.js
```

For a repository-local built image, the equivalent package entry point is
`pnpm --filter @oshun/bff reproject:personalization-index`.

## 7. Validation

- The command exits zero and its JSON reports equal `authoritativeCount`,
  `projectedCount`, and `verifiedCount`.
- `metadata.dimensions` equals `OSHUN_PERSONALIZATION_VECTOR_DIMENSIONS`,
  distance is cosine, and all three required payload-index fields are present.
- A known vector query returns the retained subject control while a tombstoned
  subject is absent. Task 13.6's isolated production-store drill automates this
  query and the empty-index reprojection control.
- BFF boot completes after the command; it independently repeats reconciliation
  and refuses malformed metadata or mismatched readback.

## 8. Fencing — NOT APPLICABLE

Nothing is promoted. There is one index and one scan fallback, and the fallback
is a read path rather than a writable replica, so there is no second writer to
reject.

## 9. Failover

Failover here is the fallback the register already declares: search reads move
to `dep:primary-store-scan`. It is a degradation and not a promotion — the scan
answers the same questions more slowly and from the authoritative rows.

Confirm the scan is actually carrying the load rather than assumed to be. The
scan carries `throttle` and `bulkhead` among its mitigations precisely because a
full-table scan serving the search surface is how a search outage becomes a
database outage.

## 10. Reconciliation

The Qdrant collection publishes nothing to an external party. Reconciliation is
still substantive: the command deletes points not present in the PostgreSQL
authority, skips tombstoned subjects, rewrites legacy payloads to the configured
embedding/schema version, and verifies the complete readback. A clean target is
therefore preferred, but a surviving stale volume is repaired rather than
trusted.

## 11. Failback

Move search reads off `dep:primary-store-scan` and back onto the index once the
document count has converged and stopped changing. Not before: an index that is
half-filled answers with a subset, and moving traffic back to it turns a slow
search into a wrong one.

Take the load off the scan promptly. It is a supporting-tier dependency doing a
core-tier job, and the longer it does it the more the estate's latency profile
looks normal for the wrong reason.

## 12. What this runbook does not cover

- The relational store the index is projected from — see
  [v1-dr-database-loss.md](v1-dr-database-loss.md) and
  [v1-dr-relational-corruption.md](v1-dr-relational-corruption.md).
- Search relevance regressions, which are not an outage.

## 13. Machine-checkable statement

<!-- oshun:runbook -->

```json
{
  "id": "runbook:v1-search-index-loss",
  "scenarioId": "gd:search-index-loss",
  "version": "2",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "workbench-platform on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:primary-store", "dep:primary-store-scan"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "search index connection credentials",
      "heldInDependencyId": "dep:search-index",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:search-index"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "the production reprojection command exits zero with authoritativeCount == projectedCount == verifiedCount",
    "collection dimensions, cosine distance and required payload indexes match the configured contract",
    "a known retained vector is queryable and a tombstoned subject is absent",
    "BFF boot repeats reconciliation before HTTP and refuses malformed metadata or mismatched readback"
  ],
  "failback": "move search reads off dep:primary-store-scan and back onto the index once the document count has converged and stopped changing"
}
```