# Deployment Rollback Runbook

**Purpose:** Provide fast, safe rollback steps for production deployments using
canary guardrails and Helm history.

## When to Trigger

- Automated guardrails fail in CI (`auto-rollback` job) or health checks degrade
  during canary.
- Error budgets breached (5xx spikes, p95 latency > 1s) within 10 minutes of
  rollout.
- Critical regression confirmed by smoke tests or on-call triage.

## Automated Path (Preferred)

1. CI auto-rollback fires when post-deploy sampling fails.
2. Pipeline triggers `helm rollback <service>-production` to the previous
   revision.
3. Incident commander verifies health:
   ```bash
   kubectl rollout status deploy/<service>-production -n production --timeout=300s
   curl -f https://<service>.lilith.app/health
   ```
4. Post in `#api-alerts` with impact, cause, and rollback timestamp.

## Manual Rollback (CLI)

Use when automation is insufficient or partial rollback is required.

```bash
SERVICE=bff
NAMESPACE=production

# Check history
helm history "$SERVICE-production" -n "$NAMESPACE"

# Roll back to previous revision
helm rollback "$SERVICE-production" 1 -n "$NAMESPACE"

# Monitor rollout
kubectl rollout status deploy/$SERVICE-production -n "$NAMESPACE" --timeout=600s

# Verify health + latency
for i in {1..5}; do curl -f https://$SERVICE.lilith.app/health && sleep 5; done
```

## Traffic Cutover After Canary Failure

- Keep canary pods scaled but remove from ingress:
  ```bash
  kubectl scale deploy/$SERVICE-canary -n production --replicas=0
  ```
- Patch ingress to route 100% to stable:
  ```bash
  kubectl patch ingress $SERVICE -n production --type='json' \
    -p='[{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service/name","value":"'"$SERVICE"'-stable"}]'
  ```

## Data / Schema Rollbacks

- If migrations are involved, validate backward compatibility before rollback.
- Use pre-deployment backups (`s3://db-backups/<service>/`) for destructive
  changes.
- For forward-only migrations, gate rollout with feature flags and shadow reads.

## Local Development Rollback (Docker Compose)

For local development environments using Docker Compose:

### Quick Rollback

```bash
# Stop all services and remove containers
docker compose -f docker-compose.dev.yml down

# Rebuild and restart (clean slate)
docker compose -f docker-compose.dev.yml build --no-cache
docker compose -f docker-compose.dev.yml up -d
```

### Rollback to Previous Container State

```bash
# View container history
docker ps -a --filter "name=lilith" --format "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.CreatedAt}}"

# Stop current containers
docker compose -f docker-compose.dev.yml stop

# Restart with previous image versions (if cached)
docker compose -f docker-compose.dev.yml up -d --no-build
```

### Database State Rollback

```bash
# Backup current database state (PostgreSQL)
docker exec lilith-postgres-1 pg_dump -U lilith lilith_dev > backup_$(date +%Y%m%d_%H%M%S).sql

# Restore from backup
cat backup_YYYYMMDD_HHMMSS.sql | docker exec -i lilith-postgres-1 psql -U lilith lilith_dev

# Reset database to initial state
docker compose -f docker-compose.dev.yml stop postgres
docker volume rm lilith_postgres_data
docker compose -f docker-compose.dev.yml up -d postgres
cat dev/init-db.sql | docker exec -i lilith-postgres-1 psql -U lilith lilith_dev
```

### Service-Specific Rollback

```bash
# Rollback single service
SERVICE=bff
docker compose -f docker-compose.dev.yml stop $SERVICE
docker compose -f docker-compose.dev.yml rm -f $SERVICE
docker compose -f docker-compose.dev.yml build $SERVICE --no-cache
docker compose -f docker-compose.dev.yml up -d $SERVICE
```

## After Action

- Capture timeline, detection source, and fix in the incident record.
- Add regression test covering the root cause.
- Validate dashboards/alerts caught the issue; close any gaps.
- Update this runbook if new steps were required.
