Metis Disaster Recovery

title: Metis Disaster Recovery owner: unassigned — see "Who owns this" below version: 2.0.0 last_reviewed: 2026-09-18 next_review: 2026-12-17 applies_to: the Metis courseware service, its Postgres database and its source object bucket#

Written for M10.18.e from the shipped code paths, with every command below executed once against the compose Postgres while writing it. The measured numbers are from that run and are named as such; where something could not be executed, this document says so rather than describing what it would have done.

Revised 2026-09-18 (M10.17). Three of the gaps version 1.0.0 recorded have moved, and one of its statements was wrong:

• backups are now scheduled and recorded — metis.tasks.durable.back_up_estate runs every backup_interval_seconds and writes a row per attempt, so the recovery point is readable rather than folklore;
• the object store half is now exercised, and it carries a constraint this document did not have: the objects must come back under the bucket name the receipts already name (see "Restore the object store");
• version 1.0.0 named the wrong buckets for blob recovery. The durable source bytes a revision's receipt points at live in metis-sources (s3_bucket_sources, M10.4) — not in metis-courses, metis-assets, metis-exports or metis-uploads, which hold derivatives, republishable exports, or objects a daily cleanup task deletes;
• the whole procedure below is now executable as one test: apps/metis/service/tests/test_estate_restore_drill_real_stack.py. Prefer running it to reading this when what you need is confidence that a restore works; read this when you are performing one.

What this covers, and what it does not#

Covered: taking a backup of the Metis database and its source bucket, restoring both into isolated copies, verifying each by hashing, and reading the recovery point off the estate itself.

Not covered, because the capability does not exist yet:

• Point-in-time recovery. archive_mode is off on the compose Postgres and wal_level is replica. A restore can only reach the state of the last dump. There is no procedure here for recovering to a chosen moment because there is no mechanism for it.
• Encrypted backups. The dumps land in object storage unencrypted, and where a key would live is an operator decision nobody has made. Treat the backup bucket as holding everything the database holds.
• A stated recovery objective. The schedule bounds the recovery point at backup_interval_seconds (default 86,400 — one day), but that is a default and not a commitment, and extrapolating the measured recovery time needs a production volume nobody has recorded (M10.14.b).

Who owns this#

Nobody, and that is a gap rather than an omission in this document. The estate has no accessibility or performance owner registry either (M10.13.f, M10.14.e), and this runbook cannot invent an on-call rota. Assign an owner before relying on it.

Find the last successful backup first#

Before anything else, learn what you are restoring to. The estate records every attempt, and a failed run is in the table beside the successes so a gap in the schedule is visible:

psql $H -d metis -c "SELECT finished_at, state, database_name, dump_bucket,
  dump_key, dump_bytes, tables, objects_copied FROM estate_backup_run
  ORDER BY finished_at DESC LIMIT 10"

The newest succeeded row is the recovery point. Its dump_key is the object to fetch from dump_bucket, and dump_sha256 is what the bytes must hash to — check that before restoring anything, not after.

If the newest success is old, or there are none, the schedule is not running: check that a beat worker is up and look for estate_backup_unavailable or estate_backup_failed in the logs. unavailable means a precondition — no pg_dump on the worker, or no backup bucket — and writes no row, which is why the log matters when the table is silent.

Prerequisites#

• pg_dump, pg_restore, createdb, dropdb on the path.
• Credentials for the compose Postgres. In development that is oshun:oshun_dev at 127.0.0.1:5432; never put the password in a committed script or in command text — export PGPASSWORD from a throwaway file and delete it afterwards.
• Enough disk for the dump. The development database was 1.1 MB compressed; production volume is unrecorded, which M10.14.b is blocked on.

Take a backup#

export PGPASSWORD=…            # from the operator's secret store, not here
H="-h 127.0.0.1 -U oshun"
pg_dump $H -Fc -f /tmp/metis-$(date +%Y%m%dT%H%M%SZ).dump metis

-Fc is the custom format, which pg_restore can restore selectively and in parallel. Measured: 204 ms for the 1.1 MB development database.

Restore into an isolated database#

Never restore over the database you are recovering from. Create a new one and compare before switching anything.

dropdb $H --if-exists metis_restore_drill
createdb $H metis_restore_drill
pg_restore $H -d metis_restore_drill --no-owner --no-privileges <dump>

Measured: 1,981 ms, total 2.2 s end to end. That is a floor for a development-sized database and says nothing about production.

--no-owner --no-privileges drops grants. On the development database that loses nothing — it carries zero role grants on public beyond the owner and PUBLIC, checked with:

psql $H -d metis -tAc "SELECT count(*) FROM information_schema.role_table_grants
  WHERE table_schema='public' AND grantee NOT IN ('oshun','PUBLIC')"

On a deployment that grants to real roles, drop those two flags and restore as a superuser, then re-run the verification below.

Verify the restore by hashing, never by counting#

A row count says two tables agree whenever they hold the same number of different rows. Hash the content instead:

for DB in metis metis_restore_drill; do
  : > "/tmp/hash-$DB.txt"
  for T in $(psql $H -d "$DB" -tAc \
      "SELECT tablename FROM pg_tables WHERE schemaname='public' ORDER BY 1"); do
    HASH=$(psql $H -d "$DB" -tAc \
      "SELECT COALESCE(md5(string_agg(x::text, E'\n' ORDER BY x::text)),'EMPTY')
       FROM \"$T\" x")
    echo "$T=$HASH" >> "/tmp/hash-$DB.txt"
  done
done
diff /tmp/hash-metis.txt /tmp/hash-metis_restore_drill.txt && echo IDENTICAL

Measured: 163 tables hashed on both sides, all 163 digests identical, with 18 tables non-empty. Check that last number — a comparison in which every table is EMPTY passes and proves nothing.

A second, independent measurement on 2026-09-18, from the drill test on a freshly provisioned stack carrying one source-to-course journey: 172 tables, all digests identical, 7 non-empty, dump 938,995 bytes. Dump 0.181 s, restore 1.790 s.

One table legitimately differs when you restore a dump this estate's own scheduled backup produced: estate_backup_run. The run row is written after the dump completes, so a dump cannot contain its own record. Every other table must match.

Tenant integrity needs no separate step: tenant_id is part of every row that was hashed.

Restore the object store#

The objects must come back under the bucket name they were stored under. This is not a preference. A revision records each blob as a BlobReceipt that carries its bucket, and BlobStore.verify reads the bucket out of the receipt rather than out of configuration — so objects restored into a differently-named bucket are objects nothing can verify, and the receipts cannot be edited to suit: they live in store_revisions, which migration 047 makes immutable outside an approved retention or legal workflow.

Measured 2026-09-18: with the objects mirrored to a second bucket and the original emptied, the reconciler reported every blob broken — and reported them broken again when pointed at the bucket that held them. Restoring into the original name made the estate reconcile.

The scheduled backup keeps the source objects under the objects/ prefix of the backup bucket, keyed exactly as they are in metis-sources. Restoring them is estate_backup.restore_sources, and it is a function rather than a shell line for two reasons. Neither mc nor the aws CLI is installed on the hosts this estate runs on (checked 2026-09-18 — which aws mc finds neither); either could be installed, but the function is the one the service's own tests exercise, and a restore procedure that is only ever run by hand is a procedure nobody has verified.

cd apps/metis/service
PYTHONPATH=src .venv/bin/python -c "
from metis.config import get_settings
from metis.services.durable_runtime import s3_client
from metis.services.estate_backup import restore_sources
settings = get_settings()
print(restore_sources(s3_client(settings), settings, settings.s3_bucket_backups))
"

It restores into s3_bucket_sources and nowhere else, refuses a bucket that is not there rather than creating one, and is idempotent — an object already present under its content-addressed key is the object the backup holds. It prints (objects_restored, bytes_restored).

Rehearsed 2026-09-18 against MinIO with one object in a backup bucket: the command above printed (1, 27), and running it again printed (0, 0).

Then verify the bytes rather than the listing — every object's length and digest, on both sides. tests/test_estate_restore_drill_real_stack.py and tests/test_estate_backup_real_stack.py do exactly that, and running them is the faster way to check the procedure still holds.

Measured: mirror 0.029 s, restore 0.045 s for 2 objects / 1,723 bytes. That is a floor at development volume and nothing more.

Reconcile after the restore#

A restored database holds outbox events nobody delivered and projections nobody has compared. Drain and reconcile before switching traffic:

# the worker task beat runs every 10 seconds; run it once against the restore
METIS_DATABASE_URL=<restored> celery -A metis.tasks.celery_app call \
  metis.tasks.durable.drain_outbox
METIS_DATABASE_URL=<restored> celery -A metis.tasks.celery_app call \
  metis.tasks.durable.reconcile_estate

reconcile_estate repairs nothing; it reports. A clean pass means the blobs, the search index, the outbox, the published exports, the grade projections and the aggregate store's own record legs all agree with the restored record. Any finding is for a person to act on before the estate is declared recovered.

Clean up#

dropdb $H --if-exists metis_restore_drill
rm -f /tmp/metis-*.dump /tmp/hash-*.txt

Leaving a restored copy on the server is how a stale database gets mistaken for the live one.

What to do when the verification fails#

Do not switch traffic. A digest mismatch means the dump, the restore or the source changed between the two hashes — take a fresh dump with the service stopped and repeat. If the mismatch persists on a quiesced source, the dump is suspect and the correct action is to escalate rather than to accept the closest copy.

Known gaps, as of the last review#

Closed since version 1.0.0: the schedule (back-up-estate, with every attempt recorded in estate_backup_run) and the object-store half (mirrored by the backup, restored and verified by the drill test).