# Runbook: Object Storage (S3 / MinIO) Backups

Owner: platform team Last reviewed: 2026-05-28

PostgreSQL backups are covered in [database-backups.md](database-backups.md).
This runbook covers the _other_ persistent store: user uploads, generated
content, recordings, and project files that live in S3 (production) or MinIO
(dev / on-prem).

Object-storage failures are different from Postgres failures:

- The data is much larger (multi-TB to PB).
- A pg_dump-equivalent isn't viable; we rely on the vendor's native versioning +
  replication.
- Recovery is per-object or per-prefix, not per-database.

## What's in scope

| Bucket / prefix                                       | Domain    | What                          | Recovery priority        |
| ----------------------------------------------------- | --------- | ----------------------------- | ------------------------ |
| `oshun-prod-uploads/users/<id>/avatars`               | yemaya    | User avatar images            | High                     |
| `oshun-prod-uploads/users/<id>/attachments`           | yemaya    | DM attachments, profile media | High                     |
| `oshun-prod-uploads/lilith/personas/<id>/audio`       | lilith    | Voice clone training audio    | Critical (consented BIO) |
| `oshun-prod-uploads/lilith/voices/<id>/samples`       | lilith    | Generated voice samples       | High                     |
| `oshun-prod-uploads/aphrodite/recordings/<stream-id>` | aphrodite | Stream VOD recordings         | High                     |
| `oshun-prod-uploads/aphrodite/uploads/<id>`           | aphrodite | Creator-uploaded clips        | High                     |
| `oshun-prod-uploads/iris/exports/<request-id>`        | iris      | GDPR / consent export bundles | Medium (regenerable)     |
| `oshun-prod-uploads/isis/outputs/<job-id>`            | isis      | Generated images / videos     | Medium (regenerable)     |
| `oshun-prod-uploads/veritas/articles/<id>/assets`     | veritas   | News article media            | High                     |

Buckets NOT in scope (no backups; intentional):

- `oshun-prod-tmp/*` — TTL'd within 24h, no recovery expected.
- `oshun-prod-test/*` — test fixtures; recreatable from
  `libs/*/testing/fixtures`.

## Backup strategy

### Production (S3)

| Layer                       | Mechanism                                                                                  | Recovery window                            |
| --------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------ |
| 1. Object versioning        | S3 versioning enabled on every in-scope bucket                                             | Indefinite (until lifecycle rule kicks in) |
| 2. Lifecycle transition     | Non-current versions → S3 Standard-IA at 30d, Glacier at 90d, expire at retention boundary | Per-bucket retention from pii-inventory.md |
| 3. Cross-region replication | Async replication to `oshun-prod-uploads-dr` in a second region                            | <15 min lag                                |
| 4. Bucket policy + ACL      | Public access blocked; deny non-TLS; require KMS encryption                                | n/a (preventive)                           |

### Dev (MinIO)

MinIO doesn't need backups — dev data is recreatable. But to keep parity with
the production runbook:

- MinIO versioning is enabled on each bucket
  (`mc version enable myminio/<bucket>`).
- No cross-region replication.
- Lifecycle rules: 7-day expiry on non-current versions.

## Recovery procedures

### Single object recovery

```bash
# Find versions of the lost object
aws s3api list-object-versions \
  --bucket oshun-prod-uploads \
  --prefix "users/$USER_ID/avatars/profile.png" \
  --query 'Versions[*].[VersionId,LastModified,IsLatest]' \
  --output table

# Copy the version you want back to the current key
aws s3api copy-object \
  --bucket oshun-prod-uploads \
  --copy-source "oshun-prod-uploads/users/$USER_ID/avatars/profile.png?versionId=$VERSION" \
  --key "users/$USER_ID/avatars/profile.png"
```

### Bulk recovery (whole prefix)

If a prefix was accidentally deleted, restore from versions:

```bash
# Get all delete-markers in the prefix
aws s3api list-object-versions \
  --bucket oshun-prod-uploads \
  --prefix "users/$USER_ID/" \
  --query 'DeleteMarkers[?IsLatest==`true`]' > /tmp/markers.json

# Delete the delete-markers, which reveals the previous versions
# as current. (Do not do this without confirming the markers list.)
jq -r '.[] | "\(.Key) \(.VersionId)"' /tmp/markers.json | \
  while read key version; do
    aws s3api delete-object \
      --bucket oshun-prod-uploads \
      --key "$key" \
      --version-id "$version"
  done
```

### Region failover (DR)

If the primary region is unavailable:

1. **Verify** the DR bucket has the data:
   ```bash
   aws s3 ls s3://oshun-prod-uploads-dr/<known-prefix>/ --region us-west-2
   ```
2. **Switch reads** by toggling the `S3_BUCKET` env var in the affected services
   (managed via Terraform; do not edit by hand).
3. **Re-establish replication** in the reverse direction once the primary region
   recovers.

The DR cutover doc lives at `docs/runbooks/dr-failover.md` (TBD).

## Validation

The platform team runs a synthetic backup test on the first Monday of each
month:

1. Upload a tagged test object to each in-scope bucket.
2. Wait 30 minutes for cross-region replication.
3. Verify the DR bucket has the object.
4. Time the recovery of a small (1 MB) object and a medium (100 MB) object.
5. Post the timings to `#platform-ops`.

If any step fails, file a SEV-2 ticket immediately.

## What's NOT covered here

- Postgres logical dumps — see [database-backups.md](database-backups.md).
- Redis snapshots — a dedicated `redis-backup.md` runbook is TBD (task #107).
- Kafka topic backups — n/a; events are reprocessable from source of truth.
- Application code / config — the git repo is the backup; deploys are
  reproducible from a commit hash.