Eve library

Launch Day Runbook

Runbook

Revision 1

On this page

Purpose: Comprehensive operational guide for Lilith Platform launch day, covering pre-launch preparation, launch execution, and post-launch monitoring.

Table of Contents#

  1. Launch Timeline
  2. Pre-Launch Checklist
  3. Launch Team & Roles
  4. Communication Plan
  5. Monitoring & Alerts
  6. Incident Response
  7. Rollback Procedures
  8. Post-Launch Review

Launch Timeline#

T-7 Days: Final Preparation Week#

Day Task Owner Status
T-7 Final load test at 2x expected traffic Engineering ☐
T-7 Security scan and remediation Security ☐
T-6 DR test - full failover drill SRE ☐
T-6 Support team briefing Support Lead ☐
T-5 Pre-scale infrastructure to launch capacity SRE ☐
T-5 Content freeze (no new deployments) Engineering ☐
T-4 Final API documentation review Technical Writer ☐
T-4 Status page and communication channels ready Comms ☐
T-3 War room setup and tested Operations ☐
T-3 On-call schedule confirmed SRE ☐
T-2 Final go/no-go meeting Leadership ☐
T-2 Smoke test in production QA ☐
T-1 Team rest day (no deployments) All ☐

Launch Day (T-0)#

text
Timeline (All times Pacific)

05:00 - War room opens, initial systems check
05:30 - All-hands ready check
06:00 - Final smoke test
06:30 - Go/no-go decision call
07:00 - DNS cutover / App store release
07:05 - First traffic arrives
07:15 - 15-minute health check
07:30 - 30-minute health check
08:00 - 1-hour health check
09:00 - Morning standup
10:00 - 3-hour checkpoint
12:00 - Midday checkpoint
14:00 - Afternoon checkpoint
16:00 - End-of-day checkpoint
18:00 - Evening transition to reduced staffing
22:00 - Night shift begins
06:00+1 - Day 2 begins

Post-Launch (T+1 to T+7)#

Day Focus Key Metrics
T+1 Stability monitoring Error rate, latency
T+2 User feedback review NPS, support tickets
T+3 Performance optimization Resource utilization
T+4 Feature flag toggles A/B test evaluation
T+5 Scaling adjustments Cost optimization
T+6 Retrospective preparation Incident review
T+7 Launch retrospective Full team review

Pre-Launch Checklist#

Infrastructure (T-48 hours)#

  • All services deployed to production
  • Database replicas in sync (<100ms lag)
  • Redis cluster healthy
  • CDN cache warmed
  • SSL certificates valid (30+ days remaining)
  • DNS TTL lowered to 60 seconds
  • Load balancers configured
  • Auto-scaling policies active
  • Backup jobs verified

Monitoring (T-48 hours)#

  • All Grafana dashboards loading
  • Alert rules active and tested
  • PagerDuty integration verified
  • Slack channels configured
  • Log aggregation working
  • APM traces flowing
  • Synthetic monitoring active

Security (T-48 hours)#

  • WAF rules active
  • DDoS protection enabled
  • Rate limiting configured
  • API keys rotated
  • Secrets vault secured
  • Security monitoring active

Support (T-48 hours)#

  • Support team trained
  • Playbooks reviewed
  • Response templates ready
  • Escalation paths tested
  • Knowledge base published
  • Status page ready

Communications (T-48 hours)#

  • Press release ready
  • Social media scheduled
  • Email campaigns queued
  • Status page configured
  • Communication templates approved

Launch Team & Roles#

War Room Structure#

text
Launch Commander
├── Technical Lead
│   ├── Backend Engineers (2)
│   ├── Frontend Engineer (1)
│   └── Mobile Engineer (1)
├── SRE/DevOps Lead
│   ├── SRE Engineers (2)
│   └── DBA (1)
├── Support Lead
│   ├── Tier 1 Support (4)
│   └── Tier 2 Support (2)
├── Communications Lead
│   └── Social Media Manager (1)
└── Product Lead
    └── QA Engineer (1)

Role Responsibilities#

Launch Commander#

  • Overall decision authority
  • Go/no-go calls
  • External stakeholder communication
  • Incident severity declaration
  • Resource allocation

Technical Lead#

  • Code-level incident response
  • Feature flag decisions
  • Hotfix approval
  • API health monitoring

SRE/DevOps Lead#

  • Infrastructure scaling decisions
  • Performance monitoring
  • Failover execution
  • Capacity management

Support Lead#

  • Customer communication
  • Ticket triage
  • Escalation management
  • Support staffing

Communications Lead#

  • Status page updates
  • Social media monitoring
  • Press inquiry handling
  • Internal announcements

Contact Sheet#

Role Primary Backup Phone Slack
Launch Commander [Name] [Name] [Phone] @launch-commander
Technical Lead [Name] [Name] [Phone] @tech-lead
SRE Lead [Name] [Name] [Phone] @sre-lead
Support Lead [Name] [Name] [Phone] @support-lead
Comms Lead [Name] [Name] [Phone] @comms-lead

Communication Plan#

Internal Communication#

Channels#

  • #launch-war-room - Primary coordination (all leads)
  • #launch-engineering - Technical discussion
  • #launch-support - Customer issues
  • #launch-status - Automated alerts and status updates

Update Cadence#

Phase Frequency Channel
Pre-launch Hourly #launch-war-room
Launch (first hour) Every 15 minutes #launch-war-room
Launch (hours 2-6) Every 30 minutes #launch-war-room
Launch (hours 6+) Hourly #launch-war-room
Incident As needed #launch-war-room + voice

Status Update Template#

text
🚀 LAUNCH STATUS UPDATE - [TIME]

📊 Key Metrics:
- Active Users: [X]
- Requests/sec: [X]
- Error Rate: [X]%
- P95 Latency: [X]ms

✅ What's Working:
- [List healthy systems]

⚠️ Issues:
- [List any issues and status]

📋 Next Actions:
- [Planned actions]

Next update: [TIME]

External Communication#

Status Page (status.lilith.app)#

Components to Track:

  • API Services
  • Web Application
  • iOS Application
  • Android Application
  • AI Services
  • Meditation Content
  • Payment Processing

Status Levels:

  • 🟢 Operational
  • 🟡 Degraded Performance
  • 🟠 Partial Outage
  • 🔴 Major Outage

Social Media Response#

For Normal Operations:

  • Monitor mentions every 15 minutes
  • Respond to positive mentions with thanks
  • Route support questions to proper channels

For Issues:

  • Acknowledge within 15 minutes
  • Use pre-approved messaging
  • Direct to status page for updates
  • Escalate media inquiries to Comms Lead

Monitoring & Alerts#

Primary Dashboards#

Dashboard URL Purpose
Launch Command Center /d/launch-command Overall system health
SLO Compliance /d/slo-compliance SLO tracking
Database Performance /d/database-perf DB metrics
AI Services /d/ai-service AI health
User Analytics /d/user-analytics User metrics

Key Metrics to Watch#

Metric Target Warning Critical
API Availability 99.9% <99.5% <99%
P95 Latency <300ms >400ms >500ms
Error Rate <0.1% >0.5% >1%
Active Users [Target] <50% target <25% target
AI Success Rate >95% <90% <85%
CPU Utilization <70% >80% >90%
Memory Utilization <75% >85% >90%
DB Connections <80% >85% >95%

Alert Severity Mapping#

Severity Response Time Who's Notified
P1 - Critical Immediate Everyone + PagerDuty
P2 - High 5 minutes Tech Lead + SRE
P3 - Medium 15 minutes SRE
P4 - Low 30 minutes Logged only

Manual Health Checks#

Run every 15 minutes during launch:

bash
# API Health Check
curl -s https://api.lilith.app/health | jq

# Auth Service
curl -s https://api.lilith.app/auth/health | jq

# Content Service
curl -s https://api.lilith.app/content/health | jq

# AI Service
curl -s https://api.lilith.app/ai/health | jq

# Database Connectivity
kubectl exec -it deploy/bff -- nc -zv postgres 5432

# Redis Connectivity
kubectl exec -it deploy/bff -- nc -zv redis 6379

Incident Response#

Incident Declaration Criteria#

P1 - Critical (Declare immediately):

  • Complete service unavailable
  • Data loss or corruption
  • Security breach
  • Payment processing failure

P2 - High (Declare within 5 minutes):

  • Major feature unavailable
  • 1% error rate

  • P95 latency >1 second
  • AI service degraded

P3 - Medium (Declare within 15 minutes):

  • Minor feature degraded
  • Single region issues
  • Performance degradation

Incident Response Procedure#

  1. Detect (0-2 minutes)

    • Alert fires or issue reported
    • Verify issue is real
  2. Declare (2-5 minutes)

    text
    /incident declare P[1/2] "Brief description"
    
    • Assign Incident Commander
    • Open incident channel
  3. Communicate (5-10 minutes)

    • Update status page
    • Notify stakeholders
    • Post in #launch-war-room
  4. Investigate (Ongoing)

    • Identify scope and impact
    • Determine root cause
    • Document timeline
  5. Mitigate (ASAP)

    • Apply fix or workaround
    • Scale if needed
    • Rollback if necessary
  6. Resolve (After mitigation)

    • Verify fix
    • Update status page
    • Notify all-clear
  7. Review (Post-incident)

    • Timeline documentation
    • Root cause analysis
    • Action items

Quick Mitigation Actions#

Issue First Action Second Action
High error rate Check recent deploys Rollback if needed
High latency Check DB connections Scale services
Service crash Check logs Restart pods
Memory pressure Check for leaks Scale horizontally
Database issues Check connections Failover if needed
AI failures Check rate limits Enable fallback mode

Rollback Procedures#

Deployment Rollback#

bash
# Identify current and previous versions
helm history bff-production -n production

# Rollback to previous version
helm rollback bff-production 1 -n production

# Verify rollback
kubectl rollout status deploy/bff-production -n production

# Confirm health
curl -s https://api.lilith.app/health

Database Rollback#

bash
# For schema changes - use migrations
npm run migrate:rollback --env=production

# For data issues - restore from backup
./scripts/restore-database.sh --point-in-time="2026-01-03 06:00:00"

Feature Flag Rollback#

bash
# Disable problematic feature
./scripts/feature-flags.sh disable NEW_FEATURE --env=production

# Verify flag is off
./scripts/feature-flags.sh status --env=production

Full Rollback Decision Tree#

text
Issue Detected
│
├─ Is service completely down?
│  ├─ YES → Immediate rollback + scale up replicas
│  └─ NO ↓
│
├─ Is error rate >1%?
│  ├─ YES → Check if related to recent deploy
│  │        ├─ YES → Rollback deploy
│  │        └─ NO → Investigate root cause
│  └─ NO ↓
│
├─ Is latency >500ms P95?
│  ├─ YES → Scale services + check database
│  └─ NO ↓
│
└─ Monitor and investigate

Post-Launch Review#

Day 1 Review (T+24 hours)#

Metrics Review:

  • Total users acquired
  • Peak concurrent users
  • Error budget consumed
  • P95 latency average
  • Support ticket volume

Questions to Answer:

  • Did we meet our launch targets?
  • What went better than expected?
  • What issues occurred?
  • What needs immediate attention?

Week 1 Retrospective (T+7 days)#

Agenda:

  1. Launch metrics summary
  2. Incident review
  3. What went well
  4. What could improve
  5. Action items

Deliverables:

  • Launch metrics report
  • Incident timeline (if any)
  • Improvement action items
  • Updated runbooks

Documentation Updates#

After launch, update:

  • Capacity planning based on actual data
  • Alert thresholds based on real patterns
  • Runbooks with lessons learned
  • Support playbooks with new issues
  • Architecture docs with changes

Appendix: Quick Commands#

Kubernetes Commands#

bash
# Check all pods
kubectl get pods -n production -o wide

# Check pod logs
kubectl logs -f deploy/bff -n production

# Scale deployment
kubectl scale deploy/bff --replicas=10 -n production

# Restart deployment
kubectl rollout restart deploy/bff -n production

Monitoring Commands#

bash
# Check current metrics
curl -s localhost:9090/api/v1/query?query=http_requests_total

# Check alerts
curl -s localhost:9090/api/v1/alerts | jq

Emergency Contacts#

text
- AWS Support: [Account number] / Premium Support
- Stripe Support: [Contact]
- CDN Support: [Contact]
- Domain Registrar: [Contact]
- PagerDuty: [Escalation policy]

Go/No-Go Checklist#

Before proceeding with launch, all items must be ✅:

Technical Go/No-Go#

  • All health checks passing
  • Error rate <0.01%
  • P95 latency <200ms
  • All services scaled to launch capacity
  • Database replication healthy
  • Backups verified within last 24 hours
  • No critical alerts firing

Operational Go/No-Go#

  • War room staffed
  • On-call schedule confirmed
  • Communication channels tested
  • Support team ready
  • Escalation paths verified
  • Rollback procedures tested

Business Go/No-Go#

  • Legal approval
  • Marketing ready
  • PR prepared
  • Executive sign-off

Final Go/No-Go Decision: ****_**** (Launch Commander)

Date/Time: ****_****


Last Updated: January 2026 Owner: SRE Team Review: Before each major launch