# Incident Response Plan

This document defines how Tara responds to production incidents, ensuring
consistent and effective handling of service disruptions.

---

## Table of Contents

1. [Incident Classification](#incident-classification)
2. [Roles and Responsibilities](#roles-and-responsibilities)
3. [Incident Response Process](#incident-response-process)
4. [Communication Protocols](#communication-protocols)
5. [Escalation Paths](#escalation-paths)
6. [On-Call Procedures](#on-call-procedures)
7. [Common Incident Playbooks](#common-incident-playbooks)
8. [Post-Incident Procedures](#post-incident-procedures)
9. [Tools and Resources](#tools-and-resources)

---

## Incident Classification

### Severity Levels

| Severity | Name     | Definition                                   | Response Time | Update Frequency |
| -------- | -------- | -------------------------------------------- | ------------- | ---------------- |
| P0       | Critical | Complete service outage or data breach       | Immediate     | Every 15 min     |
| P1       | Major    | Core feature unavailable, significant impact | 15 minutes    | Every 30 min     |
| P2       | Moderate | Partial degradation, workaround available    | 1 hour        | Every 2 hours    |
| P3       | Minor    | Cosmetic issue, minimal user impact          | 4 hours       | As needed        |

### P0 - Critical Incidents

Examples:

- Complete application unavailability
- Database corruption or data loss
- Security breach or unauthorized access
- Payment system complete failure
- Mass user credential exposure

Response expectations:

- All hands on deck
- Immediate executive notification
- Status page updated immediately
- User communication within 1 hour

### P1 - Major Incidents

Examples:

- Authentication system down
- Meditation playback completely broken
- Subscription purchases failing
- Push notifications system failure
- Significant performance degradation (>500% slower)

Response expectations:

- On-call engineer responds immediately
- Engineering lead notified within 15 minutes
- Status page updated within 15 minutes
- Additional resources pulled if needed

### P2 - Moderate Incidents

Examples:

- Single feature broken (e.g., search not working)
- Slow response times (50-200% slower)
- Intermittent errors (<5% of requests)
- Download feature broken
- Progress sync delayed

Response expectations:

- On-call engineer investigates within 1 hour
- Fix or workaround within 4 hours
- May be handled during business hours

### P3 - Minor Incidents

Examples:

- Visual/UI bugs
- Typos in content
- Non-critical feature edge cases
- Minor performance issues
- Single user reports

Response expectations:

- Tracked for next release
- Addressed during normal development
- No immediate response required

---

## Roles and Responsibilities

### Incident Commander (IC)

The IC leads the incident response and has final decision authority.

**Responsibilities:**

- Declare incident severity
- Coordinate response efforts
- Make key decisions
- Manage communication
- Declare incident resolved

**Who can be IC:**

- On-call engineer (default)
- Engineering lead
- Any senior engineer

### Technical Lead

Leads the technical investigation and resolution.

**Responsibilities:**

- Diagnose root cause
- Propose and implement fixes
- Coordinate with other engineers
- Document technical findings

### Communications Lead

Manages all internal and external communications.

**Responsibilities:**

- Update status page
- Send internal notifications
- Draft user communications
- Handle press inquiries (if any)

### Scribe

Documents the incident timeline and actions.

**Responsibilities:**

- Maintain incident timeline
- Record all decisions
- Track action items
- Prepare postmortem content

---

## Incident Response Process

### Phase 1: Detection (0-5 minutes)

```
Alert Triggered / Issue Reported
         │
         ▼
┌─────────────────────────────────┐
│ On-Call Engineer Acknowledges   │
│ - Review alert details          │
│ - Quick triage                  │
│ - Initial severity assessment   │
└─────────────────────────────────┘
         │
         ▼
   Incident Declared
```

**Actions:**

1. Acknowledge alert in monitoring system
2. Quickly assess scope and severity
3. Declare incident in #incidents channel
4. Assign initial roles (IC = on-call by default)

### Phase 2: Triage (5-15 minutes)

```
   Incident Declared
         │
         ▼
┌─────────────────────────────────┐
│ Gather Information              │
│ - Check dashboards              │
│ - Review recent deployments     │
│ - Check error logs              │
│ - Identify affected users       │
└─────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│ Confirm Severity                │
│ - P0: Escalate immediately      │
│ - P1: Notify engineering lead   │
│ - P2/P3: Continue investigation │
└─────────────────────────────────┘
```

**Actions:**

1. Open monitoring dashboards
2. Check recent changes (deployments, config)
3. Review error logs and stack traces
4. Determine blast radius
5. Confirm or adjust severity level

### Phase 3: Mitigation (15 minutes - 4 hours)

```
  Triage Complete
         │
         ▼
┌─────────────────────────────────┐
│ Stabilize the System            │
│ - Apply temporary fixes         │
│ - Rollback if necessary         │
│ - Enable feature flags          │
│ - Scale resources               │
└─────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│ Verify Mitigation               │
│ - Monitor metrics               │
│ - Test affected functionality   │
│ - Confirm user reports          │
└─────────────────────────────────┘
```

**Actions:**

1. Implement quickest path to stability
2. Prioritize user impact over root cause
3. Use established playbooks
4. Verify metrics returning to normal
5. Update status page with progress

### Phase 4: Resolution (Variable)

```
 System Stabilized
         │
         ▼
┌─────────────────────────────────┐
│ Permanent Fix (if needed)       │
│ - Develop proper solution       │
│ - Test thoroughly               │
│ - Deploy fix                    │
│ - Verify resolution             │
└─────────────────────────────────┘
         │
         ▼
 Incident Resolved
```

**Actions:**

1. Develop permanent fix (if mitigation was temporary)
2. Follow normal deployment procedures
3. Monitor for recurrence
4. Update documentation if needed

### Phase 5: Closure

```
 Incident Resolved
         │
         ▼
┌─────────────────────────────────┐
│ Wrap-Up                         │
│ - Final status update           │
│ - Internal notification         │
│ - Schedule postmortem           │
│ - Create follow-up tickets      │
└─────────────────────────────────┘
```

**Actions:**

1. Update status page to "Resolved"
2. Send final internal communication
3. Schedule postmortem (within 48 hours for P0/P1)
4. Create tickets for follow-up items
5. Thank responders

---

## Communication Protocols

### Internal Communication

#### Incident Declaration

```
@channel 🚨 INCIDENT DECLARED

Severity: P[0/1/2]
Service: [Affected service]
Impact: [Brief description]
IC: [Name]

Bridge: [Zoom/Meet link if needed]
Updates: Every [15/30] minutes

#incident-[timestamp]
```

#### Progress Updates

```
📊 INCIDENT UPDATE #[number]

Status: [Investigating / Mitigating / Monitoring]
Progress: [What's been done]
Next: [What's being tried next]
ETA: [If known, otherwise "Unknown"]

IC: [Name]
```

#### Resolution

```
✅ INCIDENT RESOLVED

Severity: P[0/1/2]
Duration: [Start] - [End] ([total time])
Impact: [Number of users, revenue impact if known]
Resolution: [Brief description]

Postmortem scheduled: [Date/Time]

Thanks to: [Responders]
```

### External Communication

#### Status Page Templates

**Investigating:**

```
Title: Investigating Issues with [Service]

We are currently investigating reports of [symptoms].
Some users may experience [impact description].

We will provide updates as we learn more.
```

**Identified:**

```
Title: Issue Identified - [Service]

We have identified the cause of [symptoms].
Our team is working on a fix.

Impact: [Description]
Workaround: [If available]

Next update in [time].
```

**Monitoring:**

```
Title: Fix Deployed - Monitoring [Service]

We have deployed a fix for [issue].
We are monitoring the system to ensure stability.

Most users should no longer experience issues.

We will provide a final update once confirmed resolved.
```

**Resolved:**

```
Title: Resolved - [Service]

The issue affecting [service] has been resolved.
All systems are operating normally.

Duration: [Start] to [End]
Impact: [Brief description]

We apologize for any inconvenience.
```

### Notification Thresholds

| Event                | Internal     | Status Page  | Users       |
| -------------------- | ------------ | ------------ | ----------- |
| P0 declared          | Immediate    | Immediate    | Within 30m  |
| P1 declared          | Immediate    | Within 15m   | If >1 hour  |
| Significant progress | Every 15-30m | Every update | As needed   |
| Resolution           | Immediate    | Immediate    | If notified |
| Postmortem scheduled | Within 24h   | N/A          | N/A         |

---

## Escalation Paths

### Engineering Escalation

```
On-Call Engineer
       │
       ├── Cannot diagnose → Engineering Lead
       │                            │
       │                            ├── Cannot resolve → CTO
       │                            │
       │                            └── Needs more resources → Pull team
       │
       └── Security issue → Security Lead → CTO → Legal
```

### Business Escalation

```
P0 Incident Declared
       │
       ├── Immediately notify → Engineering Lead
       │                              │
       ├── Within 15 minutes → Product Lead
       │                              │
       └── Within 30 minutes → Executive Team
```

### Escalation Contacts

| Role             | Primary Contact | Backup Contact | Method        |
| ---------------- | --------------- | -------------- | ------------- |
| On-Call Engineer | PagerDuty       | Phone tree     | Automated     |
| Engineering Lead | [Name]          | [Name]         | Slack + Phone |
| Product Lead     | [Name]          | [Name]         | Slack + Phone |
| Security Lead    | [Name]          | [Name]         | Slack + Phone |
| CTO              | [Name]          | [Name]         | Phone         |
| CEO              | [Name]          | N/A            | Phone         |

### When to Escalate

**Escalate to Engineering Lead when:**

- P0 or P1 incident
- Cannot diagnose within 30 minutes
- Need access not available to on-call
- Decision beyond on-call authority

**Escalate to Executive when:**

- P0 incident lasting > 1 hour
- Data breach or security incident
- Significant revenue impact
- Potential press/media involvement

---

## On-Call Procedures

### On-Call Schedule

- **Primary:** Week-long rotation, Mon 9am to Mon 9am
- **Secondary:** Backup for primary, shadow role
- **Schedule:** Maintained in PagerDuty/OpsGenie

### On-Call Responsibilities

1. **Respond to alerts within SLA**
   - P0/P1: 15 minutes (24/7)
   - P2: 1 hour (business hours)

2. **Investigate and triage**
   - Determine severity
   - Begin initial response

3. **Communicate**
   - Update #incidents channel
   - Escalate as needed

4. **Handoff**
   - Document open issues
   - Brief next on-call

### On-Call Best Practices

**DO:**

- Keep your phone charged and with you
- Test alert routing at shift start
- Acknowledge alerts promptly
- Ask for help when stuck
- Document everything

**DON'T:**

- Let alert fatigue cause missed pages
- Work beyond your capability alone
- Make major changes without review
- Forget to hand off issues

### On-Call Handoff Template

```
On-Call Handoff: [Date]

Active Issues:
- [Issue 1]: [Status, next steps]
- [Issue 2]: [Status, next steps]

Recent Incidents:
- [Date] P[X]: [Brief description, link to postmortem]

Upcoming Risks:
- [Deployment/change that might cause issues]

Notes for next on-call:
- [Any relevant information]
```

---

## Common Incident Playbooks

### Playbook: API Unresponsive

**Symptoms:**

- Health checks failing
- 5xx errors in monitoring
- Users reporting app errors

**Diagnostic Steps:**

```bash
# 1. Check service health
curl -v https://api.tara.app/health

# 2. Check container/instance status
aws ecs describe-services --cluster tara-production --services tara-api

# 3. Check recent deployments
vercel list tara-api | head -5

# 4. Check logs
aws logs tail /ecs/tara-api --since 30m

# 5. Check database connectivity
aws rds describe-db-instances --db-instance-identifier tara-production
```

**Resolution Options:**

1. Restart service
2. Rollback recent deployment
3. Scale up instances
4. Fix database connection
5. Clear cache

### Playbook: Database Connection Issues

**Symptoms:**

- Database connection timeouts
- Slow queries
- Connection pool exhausted

**Diagnostic Steps:**

```bash
# 1. Check RDS status
aws rds describe-db-instances --db-instance-identifier tara-production

# 2. Check connection count
# In psql:
SELECT count(*) FROM pg_stat_activity;

# 3. Check for long-running queries
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC;

# 4. Check instance metrics
# CloudWatch: CPUUtilization, FreeableMemory, DatabaseConnections
```

**Resolution Options:**

1. Kill long-running queries
2. Restart connection pool (app restart)
3. Scale database instance
4. Enable/adjust read replicas
5. Optimize problematic queries

### Playbook: High Error Rate

**Symptoms:**

- Error rate > 1%
- Specific endpoint failing
- Partial functionality broken

**Diagnostic Steps:**

```bash
# 1. Identify affected endpoints
# Check error tracking (Sentry/DataDog)

# 2. Check error logs
aws logs filter-log-events \
  --log-group-name /ecs/tara-api \
  --filter-pattern "ERROR"

# 3. Check recent changes
git log --oneline -10

# 4. Compare with working version
# Review deployment history
```

**Resolution Options:**

1. Rollback specific change
2. Feature flag disable
3. Fix and hotfix
4. Scale resources if load-related

### Playbook: Performance Degradation

**Symptoms:**

- Response times increased
- Users reporting slowness
- Timeouts occurring

**Diagnostic Steps:**

```bash
# 1. Check APM for slow endpoints
# DataDog/New Relic traces

# 2. Check resource utilization
# CPU, Memory, I/O metrics

# 3. Check database performance
# Slow query log, connection stats

# 4. Check cache hit rates
# Redis metrics

# 5. Check CDN/network
# CDN analytics, latency metrics
```

**Resolution Options:**

1. Scale up/out
2. Enable caching
3. Optimize queries
4. Clear problematic cache
5. Enable CDN edge caching

### Playbook: Authentication Failures

**Symptoms:**

- Users cannot log in
- Token validation failing
- Session errors

**Diagnostic Steps:**

```bash
# 1. Test auth endpoints
curl -X POST https://api.tara.app/auth/login \
  -d '{"email":"test@test.com","password":"test"}'

# 2. Check auth service logs
# Look for JWT errors, database errors

# 3. Check third-party auth
# Apple/Google auth status

# 4. Check token secrets
# Verify environment variables

# 5. Check session storage (Redis)
redis-cli ping
```

**Resolution Options:**

1. Restart auth service
2. Verify JWT secrets
3. Check third-party provider status
4. Clear session cache
5. Rollback auth changes

### Playbook: Payment Processing Failures

**Symptoms:**

- Purchases not completing
- Subscription status incorrect
- Receipt validation failing

**Diagnostic Steps:**

```bash
# 1. Check payment provider status
# RevenueCat dashboard / Stripe dashboard

# 2. Check webhook delivery
# Webhook logs in provider dashboard

# 3. Verify API keys
# Check environment configuration

# 4. Test subscription status
# Query user subscription status
```

**Resolution Options:**

1. Verify API credentials
2. Check webhook endpoints
3. Manual subscription verification
4. Contact payment provider
5. Enable backup payment flow

---

## Post-Incident Procedures

### Postmortem Requirements

| Severity | Postmortem Required | Timeline      |
| -------- | ------------------- | ------------- |
| P0       | Yes                 | Within 48h    |
| P1       | Yes                 | Within 72h    |
| P2       | Recommended         | Within 1 week |
| P3       | Optional            | If requested  |

### Postmortem Template

See
[Rollback Plan - Postmortem Template](./rollback-plan.md#postmortem-template)

### Follow-Up Actions

After every P0/P1 incident:

1. **Postmortem meeting** - Blameless review with all responders
2. **Action items** - Concrete improvements with owners and deadlines
3. **Documentation** - Update runbooks and playbooks
4. **Training** - Share learnings with broader team
5. **Metrics** - Track MTTR, MTTD improvements

### Incident Metrics

Track these metrics monthly:

| Metric                          | Definition                          |
| ------------------------------- | ----------------------------------- |
| MTTD (Mean Time to Detect)      | Time from issue start to alert      |
| MTTA (Mean Time to Acknowledge) | Time from alert to human response   |
| MTTR (Mean Time to Resolve)     | Time from issue start to resolution |
| Incident count by severity      | Number of P0, P1, P2, P3 incidents  |
| Alert noise ratio               | Actionable alerts / Total alerts    |

---

## Tools and Resources

### Monitoring Tools

| Tool        | Purpose            | URL               |
| ----------- | ------------------ | ----------------- |
| DataDog     | APM, logs, metrics | app.datadoghq.com |
| Sentry      | Error tracking     | sentry.io         |
| PagerDuty   | Alerting, on-call  | pagerduty.com     |
| Status Page | External status    | status.tara.app   |

### Access Requirements

On-call engineers need:

- [ ] AWS Console access
- [ ] Vercel access
- [ ] Database read access
- [ ] Monitoring dashboards
- [ ] PagerDuty account
- [ ] Status page admin

### Quick Links

| Resource         | Link            |
| ---------------- | --------------- |
| API Dashboard    | [Internal link] |
| Database Console | [Internal link] |
| Deployment Log   | [Internal link] |
| Runbook Index    | [Internal link] |
| On-Call Schedule | [Internal link] |

### Emergency Contacts

| Role            | Name                   | Phone             | Availability |
| --------------- | ---------------------- | ----------------- | ------------ |
| On-Call Primary | PagerDuty Primary      | PagerDuty         | 24/7         |
| Engineering     | Tara Platform Lead     | PagerDuty + Slack | 24/7 for P0  |
| Security        | Security Response Lead | PagerDuty + Slack | 24/7 for P0  |
| Executive       | Tara GM                | Phone + Email     | P0 only      |

---

_Last Updated: January 2026_

_For questions about incident response, contact ops@tara.app_
