Fighting Game · Guides & deep dives

V2 Asset Backup And Disaster Recovery

The backup plan runs every six hours with cron 15 */6 * * *.

6sections1 minread
Markdown source
On this page

This guide covers TODOS.phase-72.72.20.1.12: scheduled snapshots of Git LFS asset storage and geographic replication for disaster recovery.

Source Of Truth#

  • Manifest: V2/ue/Build/AssetPipeline/v2-asset-backup-dr.json
  • Tool: V2/ue/Tools/v2_asset_backup_dr.py
  • Validator: V2/ue/Tools/check-v2-asset-backup-dr.py
  • Scheduled workflow: .github/workflows/v2-asset-backup-dr.yml
  • Plan output: V2/ue/Saved/CI/AssetBackupDRPlan.json

Snapshot Schedule#

The backup plan runs every six hours with cron 15 */6 * * *. The plan captures versioned LFS objects, LFS lock state, a pointer inventory, and a SHA-256 checksum manifest. Source deletion is explicitly disabled; backup jobs only copy from the primary LFS bucket.

Geographic Replication#

The primary LFS backend is oshun-v2-lfs-assets in us-east-1. Each snapshot is planned to replicate to us-west-2 and eu-central-1 replica buckets using the oshun-v2-lfs-assets-{region}-backup naming convention. Replicas retain object versions, object-lock semantics, and AES256 server-side encryption.

Restore Drills#

Restore drills run from the latest snapshot pointer, copy replica content into a scratch prefix, then validate with:

bash
git lfs fsck
git lfs ls-files --all --long
python3 V2/ue/Tools/check-v2-asset-deduplication.py
python3 V2/ue/Tools/check-v2-asset-size-budget.py

The RPO target is six hours and the RTO target is four hours. Promoting a replica to production requires manual approval so a failed region cannot rewrite the active Git LFS endpoint automatically.

Planning Commands#

Generate and verify the DR plan:

bash
python3 V2/ue/Tools/v2_asset_backup_dr.py plan \
  --json-report V2/ue/Saved/CI/AssetBackupDRPlan.json

python3 V2/ue/Tools/v2_asset_backup_dr.py verify \
  --plan-json V2/ue/Saved/CI/AssetBackupDRPlan.json

The tool emits executable storage commands for the backup runner, but CI treats the plan as dry-run evidence unless storage credentials are supplied by the scheduled backup environment.

Validation Commands#

bash
python3 V2/ue/Tools/v2_asset_backup_dr.py self-test
python3 V2/ue/Tools/check-v2-asset-backup-dr.py
python3 -m json.tool V2/ue/Build/AssetPipeline/v2-asset-backup-dr.json
python3 V2/ue/Tools/check-v2-ci-workflow.py