Purpose: Comprehensive operational guide for Lilith Platform launch day, covering pre-launch preparation, launch execution, and post-launch monitoring.
Table of Contents#
- Launch Timeline
- Pre-Launch Checklist
- Launch Team & Roles
- Communication Plan
- Monitoring & Alerts
- Incident Response
- Rollback Procedures
- Post-Launch Review
Launch Timeline#
T-7 Days: Final Preparation Week#
| Day | Task | Owner | Status |
|---|---|---|---|
| T-7 | Final load test at 2x expected traffic | Engineering | ☐ |
| T-7 | Security scan and remediation | Security | ☐ |
| T-6 | DR test - full failover drill | SRE | ☐ |
| T-6 | Support team briefing | Support Lead | ☐ |
| T-5 | Pre-scale infrastructure to launch capacity | SRE | ☐ |
| T-5 | Content freeze (no new deployments) | Engineering | ☐ |
| T-4 | Final API documentation review | Technical Writer | ☐ |
| T-4 | Status page and communication channels ready | Comms | ☐ |
| T-3 | War room setup and tested | Operations | ☐ |
| T-3 | On-call schedule confirmed | SRE | ☐ |
| T-2 | Final go/no-go meeting | Leadership | ☐ |
| T-2 | Smoke test in production | QA | ☐ |
| T-1 | Team rest day (no deployments) | All | ☐ |
Launch Day (T-0)#
Timeline (All times Pacific)
05:00 - War room opens, initial systems check
05:30 - All-hands ready check
06:00 - Final smoke test
06:30 - Go/no-go decision call
07:00 - DNS cutover / App store release
07:05 - First traffic arrives
07:15 - 15-minute health check
07:30 - 30-minute health check
08:00 - 1-hour health check
09:00 - Morning standup
10:00 - 3-hour checkpoint
12:00 - Midday checkpoint
14:00 - Afternoon checkpoint
16:00 - End-of-day checkpoint
18:00 - Evening transition to reduced staffing
22:00 - Night shift begins
06:00+1 - Day 2 begins
Post-Launch (T+1 to T+7)#
| Day | Focus | Key Metrics |
|---|---|---|
| T+1 | Stability monitoring | Error rate, latency |
| T+2 | User feedback review | NPS, support tickets |
| T+3 | Performance optimization | Resource utilization |
| T+4 | Feature flag toggles | A/B test evaluation |
| T+5 | Scaling adjustments | Cost optimization |
| T+6 | Retrospective preparation | Incident review |
| T+7 | Launch retrospective | Full team review |
Pre-Launch Checklist#
Infrastructure (T-48 hours)#
- All services deployed to production
- Database replicas in sync (<100ms lag)
- Redis cluster healthy
- CDN cache warmed
- SSL certificates valid (30+ days remaining)
- DNS TTL lowered to 60 seconds
- Load balancers configured
- Auto-scaling policies active
- Backup jobs verified
Monitoring (T-48 hours)#
- All Grafana dashboards loading
- Alert rules active and tested
- PagerDuty integration verified
- Slack channels configured
- Log aggregation working
- APM traces flowing
- Synthetic monitoring active
Security (T-48 hours)#
- WAF rules active
- DDoS protection enabled
- Rate limiting configured
- API keys rotated
- Secrets vault secured
- Security monitoring active
Support (T-48 hours)#
- Support team trained
- Playbooks reviewed
- Response templates ready
- Escalation paths tested
- Knowledge base published
- Status page ready
Communications (T-48 hours)#
- Press release ready
- Social media scheduled
- Email campaigns queued
- Status page configured
- Communication templates approved
Launch Team & Roles#
War Room Structure#
Launch Commander
├── Technical Lead
│ ├── Backend Engineers (2)
│ ├── Frontend Engineer (1)
│ └── Mobile Engineer (1)
├── SRE/DevOps Lead
│ ├── SRE Engineers (2)
│ └── DBA (1)
├── Support Lead
│ ├── Tier 1 Support (4)
│ └── Tier 2 Support (2)
├── Communications Lead
│ └── Social Media Manager (1)
└── Product Lead
└── QA Engineer (1)
Role Responsibilities#
Launch Commander#
- Overall decision authority
- Go/no-go calls
- External stakeholder communication
- Incident severity declaration
- Resource allocation
Technical Lead#
- Code-level incident response
- Feature flag decisions
- Hotfix approval
- API health monitoring
SRE/DevOps Lead#
- Infrastructure scaling decisions
- Performance monitoring
- Failover execution
- Capacity management
Support Lead#
- Customer communication
- Ticket triage
- Escalation management
- Support staffing
Communications Lead#
- Status page updates
- Social media monitoring
- Press inquiry handling
- Internal announcements
Contact Sheet#
| Role | Primary | Backup | Phone | Slack |
|---|---|---|---|---|
| Launch Commander | [Name] | [Name] | [Phone] | @launch-commander |
| Technical Lead | [Name] | [Name] | [Phone] | @tech-lead |
| SRE Lead | [Name] | [Name] | [Phone] | @sre-lead |
| Support Lead | [Name] | [Name] | [Phone] | @support-lead |
| Comms Lead | [Name] | [Name] | [Phone] | @comms-lead |
Communication Plan#
Internal Communication#
Channels#
- #launch-war-room - Primary coordination (all leads)
- #launch-engineering - Technical discussion
- #launch-support - Customer issues
- #launch-status - Automated alerts and status updates
Update Cadence#
| Phase | Frequency | Channel |
|---|---|---|
| Pre-launch | Hourly | #launch-war-room |
| Launch (first hour) | Every 15 minutes | #launch-war-room |
| Launch (hours 2-6) | Every 30 minutes | #launch-war-room |
| Launch (hours 6+) | Hourly | #launch-war-room |
| Incident | As needed | #launch-war-room + voice |
Status Update Template#
🚀 LAUNCH STATUS UPDATE - [TIME]
📊 Key Metrics:
- Active Users: [X]
- Requests/sec: [X]
- Error Rate: [X]%
- P95 Latency: [X]ms
✅ What's Working:
- [List healthy systems]
⚠️ Issues:
- [List any issues and status]
📋 Next Actions:
- [Planned actions]
Next update: [TIME]
External Communication#
Status Page (status.lilith.app)#
Components to Track:
- API Services
- Web Application
- iOS Application
- Android Application
- AI Services
- Meditation Content
- Payment Processing
Status Levels:
- 🟢 Operational
- 🟡 Degraded Performance
- 🟠 Partial Outage
- 🔴 Major Outage
Social Media Response#
For Normal Operations:
- Monitor mentions every 15 minutes
- Respond to positive mentions with thanks
- Route support questions to proper channels
For Issues:
- Acknowledge within 15 minutes
- Use pre-approved messaging
- Direct to status page for updates
- Escalate media inquiries to Comms Lead
Monitoring & Alerts#
Primary Dashboards#
| Dashboard | URL | Purpose |
|---|---|---|
| Launch Command Center | /d/launch-command | Overall system health |
| SLO Compliance | /d/slo-compliance | SLO tracking |
| Database Performance | /d/database-perf | DB metrics |
| AI Services | /d/ai-service | AI health |
| User Analytics | /d/user-analytics | User metrics |
Key Metrics to Watch#
| Metric | Target | Warning | Critical |
|---|---|---|---|
| API Availability | 99.9% | <99.5% | <99% |
| P95 Latency | <300ms | >400ms | >500ms |
| Error Rate | <0.1% | >0.5% | >1% |
| Active Users | [Target] | <50% target | <25% target |
| AI Success Rate | >95% | <90% | <85% |
| CPU Utilization | <70% | >80% | >90% |
| Memory Utilization | <75% | >85% | >90% |
| DB Connections | <80% | >85% | >95% |
Alert Severity Mapping#
| Severity | Response Time | Who's Notified |
|---|---|---|
| P1 - Critical | Immediate | Everyone + PagerDuty |
| P2 - High | 5 minutes | Tech Lead + SRE |
| P3 - Medium | 15 minutes | SRE |
| P4 - Low | 30 minutes | Logged only |
Manual Health Checks#
Run every 15 minutes during launch:
# API Health Check
curl -s https://api.lilith.app/health | jq
# Auth Service
curl -s https://api.lilith.app/auth/health | jq
# Content Service
curl -s https://api.lilith.app/content/health | jq
# AI Service
curl -s https://api.lilith.app/ai/health | jq
# Database Connectivity
kubectl exec -it deploy/bff -- nc -zv postgres 5432
# Redis Connectivity
kubectl exec -it deploy/bff -- nc -zv redis 6379
Incident Response#
Incident Declaration Criteria#
P1 - Critical (Declare immediately):
- Complete service unavailable
- Data loss or corruption
- Security breach
- Payment processing failure
P2 - High (Declare within 5 minutes):
- Major feature unavailable
-
1% error rate
- P95 latency >1 second
- AI service degraded
P3 - Medium (Declare within 15 minutes):
- Minor feature degraded
- Single region issues
- Performance degradation
Incident Response Procedure#
-
Detect (0-2 minutes)
- Alert fires or issue reported
- Verify issue is real
-
Declare (2-5 minutes)
text/incident declare P[1/2] "Brief description"- Assign Incident Commander
- Open incident channel
-
Communicate (5-10 minutes)
- Update status page
- Notify stakeholders
- Post in #launch-war-room
-
Investigate (Ongoing)
- Identify scope and impact
- Determine root cause
- Document timeline
-
Mitigate (ASAP)
- Apply fix or workaround
- Scale if needed
- Rollback if necessary
-
Resolve (After mitigation)
- Verify fix
- Update status page
- Notify all-clear
-
Review (Post-incident)
- Timeline documentation
- Root cause analysis
- Action items
Quick Mitigation Actions#
| Issue | First Action | Second Action |
|---|---|---|
| High error rate | Check recent deploys | Rollback if needed |
| High latency | Check DB connections | Scale services |
| Service crash | Check logs | Restart pods |
| Memory pressure | Check for leaks | Scale horizontally |
| Database issues | Check connections | Failover if needed |
| AI failures | Check rate limits | Enable fallback mode |
Rollback Procedures#
Deployment Rollback#
# Identify current and previous versions
helm history bff-production -n production
# Rollback to previous version
helm rollback bff-production 1 -n production
# Verify rollback
kubectl rollout status deploy/bff-production -n production
# Confirm health
curl -s https://api.lilith.app/health
Database Rollback#
# For schema changes - use migrations
npm run migrate:rollback --env=production
# For data issues - restore from backup
./scripts/restore-database.sh --point-in-time="2026-01-03 06:00:00"
Feature Flag Rollback#
# Disable problematic feature
./scripts/feature-flags.sh disable NEW_FEATURE --env=production
# Verify flag is off
./scripts/feature-flags.sh status --env=production
Full Rollback Decision Tree#
Issue Detected
│
├─ Is service completely down?
│ ├─ YES → Immediate rollback + scale up replicas
│ └─ NO ↓
│
├─ Is error rate >1%?
│ ├─ YES → Check if related to recent deploy
│ │ ├─ YES → Rollback deploy
│ │ └─ NO → Investigate root cause
│ └─ NO ↓
│
├─ Is latency >500ms P95?
│ ├─ YES → Scale services + check database
│ └─ NO ↓
│
└─ Monitor and investigate
Post-Launch Review#
Day 1 Review (T+24 hours)#
Metrics Review:
- Total users acquired
- Peak concurrent users
- Error budget consumed
- P95 latency average
- Support ticket volume
Questions to Answer:
- Did we meet our launch targets?
- What went better than expected?
- What issues occurred?
- What needs immediate attention?
Week 1 Retrospective (T+7 days)#
Agenda:
- Launch metrics summary
- Incident review
- What went well
- What could improve
- Action items
Deliverables:
- Launch metrics report
- Incident timeline (if any)
- Improvement action items
- Updated runbooks
Documentation Updates#
After launch, update:
- Capacity planning based on actual data
- Alert thresholds based on real patterns
- Runbooks with lessons learned
- Support playbooks with new issues
- Architecture docs with changes
Appendix: Quick Commands#
Kubernetes Commands#
# Check all pods
kubectl get pods -n production -o wide
# Check pod logs
kubectl logs -f deploy/bff -n production
# Scale deployment
kubectl scale deploy/bff --replicas=10 -n production
# Restart deployment
kubectl rollout restart deploy/bff -n production
Monitoring Commands#
# Check current metrics
curl -s localhost:9090/api/v1/query?query=http_requests_total
# Check alerts
curl -s localhost:9090/api/v1/alerts | jq
Emergency Contacts#
- AWS Support: [Account number] / Premium Support
- Stripe Support: [Contact]
- CDN Support: [Contact]
- Domain Registrar: [Contact]
- PagerDuty: [Escalation policy]
Go/No-Go Checklist#
Before proceeding with launch, all items must be ✅:
Technical Go/No-Go#
- All health checks passing
- Error rate <0.01%
- P95 latency <200ms
- All services scaled to launch capacity
- Database replication healthy
- Backups verified within last 24 hours
- No critical alerts firing
Operational Go/No-Go#
- War room staffed
- On-call schedule confirmed
- Communication channels tested
- Support team ready
- Escalation paths verified
- Rollback procedures tested
Business Go/No-Go#
- Legal approval
- Marketing ready
- PR prepared
- Executive sign-off
Final Go/No-Go Decision: ****_**** (Launch Commander)
Date/Time: ****_****
Last Updated: January 2026 Owner: SRE Team Review: Before each major launch