Runbook: Object Storage (S3 / MinIO) Backups

Owner: platform team Last reviewed: 2026-05-28

PostgreSQL backups are covered in database-backups.md. This runbook covers the other persistent store: user uploads, generated content, recordings, and project files that live in S3 (production) or MinIO (dev / on-prem).

Object-storage failures are different from Postgres failures:

• The data is much larger (multi-TB to PB).
• A pg_dump-equivalent isn't viable; we rely on the vendor's native versioning + replication.
• Recovery is per-object or per-prefix, not per-database.

What's in scope#

Buckets NOT in scope (no backups; intentional):

• oshun-prod-tmp/* — TTL'd within 24h, no recovery expected.
• oshun-prod-test/* — test fixtures; recreatable from libs/*/testing/fixtures.

Backup strategy#

Production (S3)#

Dev (MinIO)#

MinIO doesn't need backups — dev data is recreatable. But to keep parity with the production runbook:

• MinIO versioning is enabled on each bucket (mc version enable myminio/<bucket>).
• No cross-region replication.
• Lifecycle rules: 7-day expiry on non-current versions.

Recovery procedures#

Single object recovery#

# Find versions of the lost object
aws s3api list-object-versions \
  --bucket oshun-prod-uploads \
  --prefix "users/$USER_ID/avatars/profile.png" \
  --query 'Versions[*].[VersionId,LastModified,IsLatest]' \
  --output table

# Copy the version you want back to the current key
aws s3api copy-object \
  --bucket oshun-prod-uploads \
  --copy-source "oshun-prod-uploads/users/$USER_ID/avatars/profile.png?versionId=$VERSION" \
  --key "users/$USER_ID/avatars/profile.png"

Bulk recovery (whole prefix)#

If a prefix was accidentally deleted, restore from versions:

# Get all delete-markers in the prefix
aws s3api list-object-versions \
  --bucket oshun-prod-uploads \
  --prefix "users/$USER_ID/" \
  --query 'DeleteMarkers[?IsLatest==`true`]' > /tmp/markers.json

# Delete the delete-markers, which reveals the previous versions
# as current. (Do not do this without confirming the markers list.)
jq -r '.[] | "\(.Key) \(.VersionId)"' /tmp/markers.json | \
  while read key version; do
    aws s3api delete-object \
      --bucket oshun-prod-uploads \
      --key "$key" \
      --version-id "$version"
  done

Region failover (DR)#

If the primary region is unavailable:

1. Verify the DR bucket has the data:
   bashCopy

   aws s3 ls s3://oshun-prod-uploads-dr/<known-prefix>/ --region us-west-2
   

2. Switch reads by toggling the S3_BUCKET env var in the affected services (managed via Terraform; do not edit by hand).
3. Re-establish replication in the reverse direction once the primary region recovers.

The DR cutover doc lives at docs/runbooks/dr-failover.md (TBD).

Validation#

The platform team runs a synthetic backup test on the first Monday of each month:

1. Upload a tagged test object to each in-scope bucket.
2. Wait 30 minutes for cross-region replication.
3. Verify the DR bucket has the object.
4. Time the recovery of a small (1 MB) object and a medium (100 MB) object.
5. Post the timings to #platform-ops.

If any step fails, file a SEV-2 ticket immediately.

What's NOT covered here#

• Postgres logical dumps — see database-backups.md.
• Redis snapshots — a dedicated redis-backup.md runbook is TBD (task #107).
• Kafka topic backups — n/a; events are reprocessable from source of truth.
• Application code / config — the git repo is the backup; deploys are reproducible from a commit hash.