# Launch Day Runbook

**Purpose**: Comprehensive operational guide for Lilith Platform launch day,
covering pre-launch preparation, launch execution, and post-launch monitoring.

## Table of Contents

1. [Launch Timeline](#launch-timeline)
2. [Pre-Launch Checklist](#pre-launch-checklist)
3. [Launch Team & Roles](#launch-team--roles)
4. [Communication Plan](#communication-plan)
5. [Monitoring & Alerts](#monitoring--alerts)
6. [Incident Response](#incident-response)
7. [Rollback Procedures](#rollback-procedures)
8. [Post-Launch Review](#post-launch-review)

---

## Launch Timeline

### T-7 Days: Final Preparation Week

| Day | Task                                         | Owner            | Status |
| --- | -------------------------------------------- | ---------------- | ------ |
| T-7 | Final load test at 2x expected traffic       | Engineering      | ☐      |
| T-7 | Security scan and remediation                | Security         | ☐      |
| T-6 | DR test - full failover drill                | SRE              | ☐      |
| T-6 | Support team briefing                        | Support Lead     | ☐      |
| T-5 | Pre-scale infrastructure to launch capacity  | SRE              | ☐      |
| T-5 | Content freeze (no new deployments)          | Engineering      | ☐      |
| T-4 | Final API documentation review               | Technical Writer | ☐      |
| T-4 | Status page and communication channels ready | Comms            | ☐      |
| T-3 | War room setup and tested                    | Operations       | ☐      |
| T-3 | On-call schedule confirmed                   | SRE              | ☐      |
| T-2 | Final go/no-go meeting                       | Leadership       | ☐      |
| T-2 | Smoke test in production                     | QA               | ☐      |
| T-1 | Team rest day (no deployments)               | All              | ☐      |

### Launch Day (T-0)

```
Timeline (All times Pacific)

05:00 - War room opens, initial systems check
05:30 - All-hands ready check
06:00 - Final smoke test
06:30 - Go/no-go decision call
07:00 - DNS cutover / App store release
07:05 - First traffic arrives
07:15 - 15-minute health check
07:30 - 30-minute health check
08:00 - 1-hour health check
09:00 - Morning standup
10:00 - 3-hour checkpoint
12:00 - Midday checkpoint
14:00 - Afternoon checkpoint
16:00 - End-of-day checkpoint
18:00 - Evening transition to reduced staffing
22:00 - Night shift begins
06:00+1 - Day 2 begins
```

### Post-Launch (T+1 to T+7)

| Day | Focus                     | Key Metrics          |
| --- | ------------------------- | -------------------- |
| T+1 | Stability monitoring      | Error rate, latency  |
| T+2 | User feedback review      | NPS, support tickets |
| T+3 | Performance optimization  | Resource utilization |
| T+4 | Feature flag toggles      | A/B test evaluation  |
| T+5 | Scaling adjustments       | Cost optimization    |
| T+6 | Retrospective preparation | Incident review      |
| T+7 | Launch retrospective      | Full team review     |

---

## Pre-Launch Checklist

### Infrastructure (T-48 hours)

- [ ] All services deployed to production
- [ ] Database replicas in sync (<100ms lag)
- [ ] Redis cluster healthy
- [ ] CDN cache warmed
- [ ] SSL certificates valid (30+ days remaining)
- [ ] DNS TTL lowered to 60 seconds
- [ ] Load balancers configured
- [ ] Auto-scaling policies active
- [ ] Backup jobs verified

### Monitoring (T-48 hours)

- [ ] All Grafana dashboards loading
- [ ] Alert rules active and tested
- [ ] PagerDuty integration verified
- [ ] Slack channels configured
- [ ] Log aggregation working
- [ ] APM traces flowing
- [ ] Synthetic monitoring active

### Security (T-48 hours)

- [ ] WAF rules active
- [ ] DDoS protection enabled
- [ ] Rate limiting configured
- [ ] API keys rotated
- [ ] Secrets vault secured
- [ ] Security monitoring active

### Support (T-48 hours)

- [ ] Support team trained
- [ ] Playbooks reviewed
- [ ] Response templates ready
- [ ] Escalation paths tested
- [ ] Knowledge base published
- [ ] Status page ready

### Communications (T-48 hours)

- [ ] Press release ready
- [ ] Social media scheduled
- [ ] Email campaigns queued
- [ ] Status page configured
- [ ] Communication templates approved

---

## Launch Team & Roles

### War Room Structure

```
Launch Commander
├── Technical Lead
│   ├── Backend Engineers (2)
│   ├── Frontend Engineer (1)
│   └── Mobile Engineer (1)
├── SRE/DevOps Lead
│   ├── SRE Engineers (2)
│   └── DBA (1)
├── Support Lead
│   ├── Tier 1 Support (4)
│   └── Tier 2 Support (2)
├── Communications Lead
│   └── Social Media Manager (1)
└── Product Lead
    └── QA Engineer (1)
```

### Role Responsibilities

#### Launch Commander

- Overall decision authority
- Go/no-go calls
- External stakeholder communication
- Incident severity declaration
- Resource allocation

#### Technical Lead

- Code-level incident response
- Feature flag decisions
- Hotfix approval
- API health monitoring

#### SRE/DevOps Lead

- Infrastructure scaling decisions
- Performance monitoring
- Failover execution
- Capacity management

#### Support Lead

- Customer communication
- Ticket triage
- Escalation management
- Support staffing

#### Communications Lead

- Status page updates
- Social media monitoring
- Press inquiry handling
- Internal announcements

### Contact Sheet

| Role             | Primary | Backup | Phone   | Slack             |
| ---------------- | ------- | ------ | ------- | ----------------- |
| Launch Commander | [Name]  | [Name] | [Phone] | @launch-commander |
| Technical Lead   | [Name]  | [Name] | [Phone] | @tech-lead        |
| SRE Lead         | [Name]  | [Name] | [Phone] | @sre-lead         |
| Support Lead     | [Name]  | [Name] | [Phone] | @support-lead     |
| Comms Lead       | [Name]  | [Name] | [Phone] | @comms-lead       |

---

## Communication Plan

### Internal Communication

#### Channels

- **#launch-war-room** - Primary coordination (all leads)
- **#launch-engineering** - Technical discussion
- **#launch-support** - Customer issues
- **#launch-status** - Automated alerts and status updates

#### Update Cadence

| Phase               | Frequency        | Channel                  |
| ------------------- | ---------------- | ------------------------ |
| Pre-launch          | Hourly           | #launch-war-room         |
| Launch (first hour) | Every 15 minutes | #launch-war-room         |
| Launch (hours 2-6)  | Every 30 minutes | #launch-war-room         |
| Launch (hours 6+)   | Hourly           | #launch-war-room         |
| Incident            | As needed        | #launch-war-room + voice |

#### Status Update Template

```
🚀 LAUNCH STATUS UPDATE - [TIME]

📊 Key Metrics:
- Active Users: [X]
- Requests/sec: [X]
- Error Rate: [X]%
- P95 Latency: [X]ms

✅ What's Working:
- [List healthy systems]

⚠️ Issues:
- [List any issues and status]

📋 Next Actions:
- [Planned actions]

Next update: [TIME]
```

### External Communication

#### Status Page (status.lilith.app)

**Components to Track**:

- API Services
- Web Application
- iOS Application
- Android Application
- AI Services
- Meditation Content
- Payment Processing

**Status Levels**:

- 🟢 Operational
- 🟡 Degraded Performance
- 🟠 Partial Outage
- 🔴 Major Outage

#### Social Media Response

**For Normal Operations**:

- Monitor mentions every 15 minutes
- Respond to positive mentions with thanks
- Route support questions to proper channels

**For Issues**:

- Acknowledge within 15 minutes
- Use pre-approved messaging
- Direct to status page for updates
- Escalate media inquiries to Comms Lead

---

## Monitoring & Alerts

### Primary Dashboards

| Dashboard             | URL               | Purpose               |
| --------------------- | ----------------- | --------------------- |
| Launch Command Center | /d/launch-command | Overall system health |
| SLO Compliance        | /d/slo-compliance | SLO tracking          |
| Database Performance  | /d/database-perf  | DB metrics            |
| AI Services           | /d/ai-service     | AI health             |
| User Analytics        | /d/user-analytics | User metrics          |

### Key Metrics to Watch

| Metric             | Target   | Warning     | Critical    |
| ------------------ | -------- | ----------- | ----------- |
| API Availability   | 99.9%    | <99.5%      | <99%        |
| P95 Latency        | <300ms   | >400ms      | >500ms      |
| Error Rate         | <0.1%    | >0.5%       | >1%         |
| Active Users       | [Target] | <50% target | <25% target |
| AI Success Rate    | >95%     | <90%        | <85%        |
| CPU Utilization    | <70%     | >80%        | >90%        |
| Memory Utilization | <75%     | >85%        | >90%        |
| DB Connections     | <80%     | >85%        | >95%        |

### Alert Severity Mapping

| Severity      | Response Time | Who's Notified       |
| ------------- | ------------- | -------------------- |
| P1 - Critical | Immediate     | Everyone + PagerDuty |
| P2 - High     | 5 minutes     | Tech Lead + SRE      |
| P3 - Medium   | 15 minutes    | SRE                  |
| P4 - Low      | 30 minutes    | Logged only          |

### Manual Health Checks

Run every 15 minutes during launch:

```bash
# API Health Check
curl -s https://api.lilith.app/health | jq

# Auth Service
curl -s https://api.lilith.app/auth/health | jq

# Content Service
curl -s https://api.lilith.app/content/health | jq

# AI Service
curl -s https://api.lilith.app/ai/health | jq

# Database Connectivity
kubectl exec -it deploy/bff -- nc -zv postgres 5432

# Redis Connectivity
kubectl exec -it deploy/bff -- nc -zv redis 6379
```

---

## Incident Response

### Incident Declaration Criteria

**P1 - Critical** (Declare immediately):

- Complete service unavailable
- Data loss or corruption
- Security breach
- Payment processing failure

**P2 - High** (Declare within 5 minutes):

- Major feature unavailable
- > 1% error rate
- P95 latency >1 second
- AI service degraded

**P3 - Medium** (Declare within 15 minutes):

- Minor feature degraded
- Single region issues
- Performance degradation

### Incident Response Procedure

1. **Detect** (0-2 minutes)
   - Alert fires or issue reported
   - Verify issue is real

2. **Declare** (2-5 minutes)

   ```
   /incident declare P[1/2] "Brief description"
   ```

   - Assign Incident Commander
   - Open incident channel

3. **Communicate** (5-10 minutes)
   - Update status page
   - Notify stakeholders
   - Post in #launch-war-room

4. **Investigate** (Ongoing)
   - Identify scope and impact
   - Determine root cause
   - Document timeline

5. **Mitigate** (ASAP)
   - Apply fix or workaround
   - Scale if needed
   - Rollback if necessary

6. **Resolve** (After mitigation)
   - Verify fix
   - Update status page
   - Notify all-clear

7. **Review** (Post-incident)
   - Timeline documentation
   - Root cause analysis
   - Action items

### Quick Mitigation Actions

| Issue           | First Action         | Second Action        |
| --------------- | -------------------- | -------------------- |
| High error rate | Check recent deploys | Rollback if needed   |
| High latency    | Check DB connections | Scale services       |
| Service crash   | Check logs           | Restart pods         |
| Memory pressure | Check for leaks      | Scale horizontally   |
| Database issues | Check connections    | Failover if needed   |
| AI failures     | Check rate limits    | Enable fallback mode |

---

## Rollback Procedures

### Deployment Rollback

```bash
# Identify current and previous versions
helm history bff-production -n production

# Rollback to previous version
helm rollback bff-production 1 -n production

# Verify rollback
kubectl rollout status deploy/bff-production -n production

# Confirm health
curl -s https://api.lilith.app/health
```

### Database Rollback

```bash
# For schema changes - use migrations
npm run migrate:rollback --env=production

# For data issues - restore from backup
./scripts/restore-database.sh --point-in-time="2026-01-03 06:00:00"
```

### Feature Flag Rollback

```bash
# Disable problematic feature
./scripts/feature-flags.sh disable NEW_FEATURE --env=production

# Verify flag is off
./scripts/feature-flags.sh status --env=production
```

### Full Rollback Decision Tree

```
Issue Detected
│
├─ Is service completely down?
│  ├─ YES → Immediate rollback + scale up replicas
│  └─ NO ↓
│
├─ Is error rate >1%?
│  ├─ YES → Check if related to recent deploy
│  │        ├─ YES → Rollback deploy
│  │        └─ NO → Investigate root cause
│  └─ NO ↓
│
├─ Is latency >500ms P95?
│  ├─ YES → Scale services + check database
│  └─ NO ↓
│
└─ Monitor and investigate
```

---

## Post-Launch Review

### Day 1 Review (T+24 hours)

**Metrics Review**:

- Total users acquired
- Peak concurrent users
- Error budget consumed
- P95 latency average
- Support ticket volume

**Questions to Answer**:

- Did we meet our launch targets?
- What went better than expected?
- What issues occurred?
- What needs immediate attention?

### Week 1 Retrospective (T+7 days)

**Agenda**:

1. Launch metrics summary
2. Incident review
3. What went well
4. What could improve
5. Action items

**Deliverables**:

- Launch metrics report
- Incident timeline (if any)
- Improvement action items
- Updated runbooks

### Documentation Updates

After launch, update:

- [ ] Capacity planning based on actual data
- [ ] Alert thresholds based on real patterns
- [ ] Runbooks with lessons learned
- [ ] Support playbooks with new issues
- [ ] Architecture docs with changes

---

## Appendix: Quick Commands

### Kubernetes Commands

```bash
# Check all pods
kubectl get pods -n production -o wide

# Check pod logs
kubectl logs -f deploy/bff -n production

# Scale deployment
kubectl scale deploy/bff --replicas=10 -n production

# Restart deployment
kubectl rollout restart deploy/bff -n production
```

### Monitoring Commands

```bash
# Check current metrics
curl -s localhost:9090/api/v1/query?query=http_requests_total

# Check alerts
curl -s localhost:9090/api/v1/alerts | jq
```

### Emergency Contacts

```
- AWS Support: [Account number] / Premium Support
- Stripe Support: [Contact]
- CDN Support: [Contact]
- Domain Registrar: [Contact]
- PagerDuty: [Escalation policy]
```

---

## Go/No-Go Checklist

Before proceeding with launch, all items must be ✅:

### Technical Go/No-Go

- [ ] All health checks passing
- [ ] Error rate <0.01%
- [ ] P95 latency <200ms
- [ ] All services scaled to launch capacity
- [ ] Database replication healthy
- [ ] Backups verified within last 24 hours
- [ ] No critical alerts firing

### Operational Go/No-Go

- [ ] War room staffed
- [ ] On-call schedule confirmed
- [ ] Communication channels tested
- [ ] Support team ready
- [ ] Escalation paths verified
- [ ] Rollback procedures tested

### Business Go/No-Go

- [ ] Legal approval
- [ ] Marketing ready
- [ ] PR prepared
- [ ] Executive sign-off

**Final Go/No-Go Decision**: **\*\*\*\***\_**\*\*\*\*** (Launch Commander)

**Date/Time**: **\*\*\*\***\_**\*\*\*\***

---

_Last Updated: January 2026_ _Owner: SRE Team_ _Review: Before each major
launch_
