Disciplines · Reference

Rollback Procedures

Most domains use Prisma for database migrations:

10sections2 minread
Markdown source
On this page

This document describes rollback procedures for the Oshun monorepo, covering ECS deployments, database migrations, infrastructure changes, and the historical Lilith/Yemaya migration.

Quick Reference#

Scenario Action Section
ECS Deployment Redeploy previous task def Section 1
Database Migration Rollback via Prisma/Knex Section 2
Service Code Rollback Revert specific commits Section 3
Infrastructure Docker/Terraform rollback Section 4
Historical Migration Restore Lilith/Yemaya bundles Section 5

1. ECS Deployment Rollback#

Revert to Previous Task Definition#

bash
# List recent task definitions for a service
aws ecs list-task-definitions --family-prefix oshun-tara-api --sort DESC --max-items 5

# Update service to use previous task definition
aws ecs update-service \
  --cluster oshun-production \
  --service tara-api \
  --task-definition oshun-tara-api:<previous-revision>

# Monitor the deployment
aws ecs describe-services --cluster oshun-production --services tara-api \
  --query 'services[0].deployments'

Force New Deployment (Same Version)#

bash
aws ecs update-service \
  --cluster oshun-production \
  --service tara-api \
  --force-new-deployment

Check Service Health#

bash
# View service events
aws ecs describe-services --cluster oshun-production --services tara-api \
  --query 'services[0].events[:5]'

# View running tasks
aws ecs list-tasks --cluster oshun-production --service-name tara-api

2. Database Rollback#

Prisma Migration Rollback#

Most domains use Prisma for database migrations:

bash
cd libs/{domain}/database

# List migrations
npx prisma migrate status

# Mark a migration as rolled back
npx prisma migrate resolve --rolled-back <migration-name>

# Full database reset (WARNING: destroys data)
npx prisma migrate reset

Knex Migration Rollback (Lilith)#

Lilith uses Knex for migrations:

bash
cd libs/lilith/database

# Rollback last migration
npx knex migrate:rollback

# Rollback all migrations
npx knex migrate:rollback --all

PostgreSQL Backup/Restore#

For critical rollbacks, restore from database backups:

bash
# Create a backup before risky operations
pg_dump -h localhost -U oshun -d tara > tara-backup-$(date +%Y%m%d).sql

# Restore from backup
psql -h localhost -U oshun -d tara < tara-backup-20260214.sql

3. Service Code Rollback#

Revert Specific Commits#

bash
# Find the commits to revert
git log --oneline -20

# Revert a specific commit
git revert <commit-hash>

# Revert a range of commits
git revert <oldest-commit>^..<newest-commit>

Restore Specific Files#

bash
# Restore a specific file from a known-good commit
git checkout <good-commit-hash> -- path/to/file

# Restore an entire directory
git checkout <good-commit-hash> -- apps/tara/api/

4. Infrastructure Rollback#

Docker Compose (Development)#

bash
# Stop all containers
docker compose -f docker/docker-compose.dev.yml down

# Remove volumes if needed (WARNING: destroys data)
docker compose -f docker/docker-compose.dev.yml down -v

# Restart fresh
docker compose -f docker/docker-compose.dev.yml up -d

Terraform#

bash
# Review what would change
cd infra/terraform/tara-api
terraform plan

# Rollback to previous state (if state is saved)
terraform apply -target=<resource> -var-file=<previous-vars>

5. Historical: Lilith/Yemaya Migration Rollback#

Note: This section documents rollback procedures from the original Lilith/Yemaya repository consolidation into the Oshun monorepo (January 2026). These bundles are kept for historical reference and emergency recovery.

Backup Locations#

Backup Location Size Created
Lilith Bundle /home/ubuntu/oshun/backups/lilith-pre-migration-20260110.bundle 47MB 2026-01-10
Yemaya Bundle /home/ubuntu/oshun/backups/yemaya-pre-migration-20260110.bundle 13MB 2026-01-10

Restore from Bundles#

bash
# Verify bundle integrity
git bundle verify /home/ubuntu/oshun/backups/lilith-pre-migration-20260110.bundle
git bundle verify /home/ubuntu/oshun/backups/yemaya-pre-migration-20260110.bundle

# Clone from bundle into a separate directory
mkdir -p /home/ubuntu/oshun-restored
git clone /home/ubuntu/oshun/backups/lilith-pre-migration-20260110.bundle /home/ubuntu/oshun-restored/lilith
git clone /home/ubuntu/oshun/backups/yemaya-pre-migration-20260110.bundle /home/ubuntu/oshun-restored/yemaya

6. Verification Checklist#

After any rollback, verify the following:

Code Verification#

  • git status shows clean working directory
  • git log -1 shows expected commit
  • All expected files are present

Build Verification#

bash
pnpm install
pnpm typecheck
pnpm lint
pnpm test

Service Verification#

  • All services start without errors
  • Health checks pass (/health endpoint returns 200)
  • API endpoints respond correctly
  • Database connections work
  • Redis connections work

Data Verification#

  • Database schemas match expected state
  • No data loss occurred
  • Foreign key constraints are valid

7. Rollback Decision Matrix#

Symptom Severity Action
Build fails Medium Revert commits (Section 3)
Tests fail Medium Service code rollback (Section 3)
Services don't start High ECS rollback (Section 1) + investigate
Database errors Critical Database rollback (Section 2)
Data corruption Critical DB restore from backup + code rollback
Production outage Critical ECS rollback + infrastructure rollback

8. Post-Rollback Actions#

After completing a rollback:

  1. Document the Issue

    • What failed?
    • When did it fail?
    • What was the error message?
    • What rollback procedure was used?

  2. Create an Incident Report

    • Timeline of events
    • Root cause analysis
    • Impact assessment
    • Prevention measures

  3. Communicate

    • Notify stakeholders
    • Update status page if applicable
    • Send team notification

Revision History#

Version Date Author Changes
1.0 2026-01-10 Development Team Initial version (migration-focused)
1.1 2026-02-14 Development Team Generalized for ongoing use; added ECS procedures