# OSHUN On-Call Runbook

> Incident response procedures and escalation paths for the OSHUN platform.

---

## Severity Levels

| Level  | Definition                                                | Response Time     | Escalation     |
| ------ | --------------------------------------------------------- | ----------------- | -------------- |
| **P0** | Service outage affecting all users (app crash, auth down) | 5 minutes         | Immediate page |
| **P1** | Major degradation affecting >10% of users                 | 15 minutes        | Page on-call   |
| **P2** | Partial degradation, single domain or feature impacted    | 1 hour            | Slack alert    |
| **P3** | Minor issue, workaround available, no user-visible impact | 4 hours           | Slack alert    |
| **P4** | Informational, no immediate action needed                 | Next business day | Slack channel  |

---

## Alert Response Procedures

### High Crash Rate (crash-rate-high)

**Trigger**: Crash-free sessions rate < 99.5% over 15 minutes.

**Steps**:

1. Check crash reporting dashboard for the top crash signature.
2. Identify the affected release version(s) and platform(s).
3. If crash is in the latest release:
   - Check if a rollback OTA update is available.
   - If OTA is not possible, prepare an emergency native build.
4. If crash is in a domain module (Tara, Arete, Veritas, Nyx, Nisaba):
   - Disable the domain module via feature flag if possible.
   - Notify the domain team lead.
5. Post status update in #oshun-incidents.

### Fatal Crash Spike (fatal-crash-spike)

**Trigger**: >10 fatal crashes in 5 minutes.

**Steps**:

1. Immediately check if a new release was deployed in the last hour.
2. If yes: initiate rollback (OTA or app store if needed).
3. If no: check for backend dependency failures that could cause client crashes.
4. Check device/OS distribution — may be platform-specific.
5. Escalate to P0 if affecting >1% of daily active users.

### High BFF Error Rate (bff-error-rate-high)

**Trigger**: BFF 5xx error rate > 5% over 10 minutes.

**Steps**:

1. Check BFF logs for the error pattern:
   ```bash
   docker logs oshun-bff --tail 200 | grep "ERROR"
   ```
2. Check if the error is from a specific route or all routes.
3. If route-specific: check the downstream domain API health.
4. If global: check BFF process health (memory, CPU, event loop).
5. If domain API is down:
   - Enable graceful degradation (cached/fallback responses).
   - Notify domain team.
6. If BFF itself is failing:
   - Check recent deployments.
   - Restart BFF pods/containers.
   - If persists, rollback to last known good deployment.

### Critical BFF Latency (bff-latency-critical)

**Trigger**: BFF P95 response time > 5 seconds over 10 minutes.

**Steps**:

1. Check if latency is from a specific route or global.
2. If route-specific:
   - Check the downstream domain API response times.
   - Check if caching is functioning (Redis health).
3. If global:
   - Check database connection pool health.
   - Check for event loop blocking (long-running synchronous operations).
   - Check memory usage for potential GC pressure.
4. Temporary mitigations:
   - Increase cache TTLs to reduce backend load.
   - Enable request shedding for non-critical endpoints.

### Slow Cold Start (startup-critical)

**Trigger**: P95 cold start time > 8 seconds over 15 minutes.

**Steps**:

1. Check if this coincides with a new release.
2. Check JS bundle size — a large bundle increase causes cold start regression.
3. Check if third-party SDKs were added or updated.
4. If caused by new release: prepare hotfix to reduce bundle size.
5. If not release-related: check if device distribution shifted (more low-end
   devices).

### Domain Unavailable (domain-unavailable)

**Trigger**: Domain API returns errors for 100% of requests over 5 minutes.

**Steps**:

1. Identify which domain is affected.
2. Check domain service health directly:
   ```bash
   curl -s https://<domain-api>/health
   ```
3. Check if the issue is network/DNS or application-level.
4. Enable fallback UI in the shell (offline cards, cached content).
5. Notify domain team lead immediately.
6. Post status update in #oshun-incidents with affected domain.

### Error Boundary Spike (error-boundary-spike)

**Trigger**: >50 error boundary triggers in 10 minutes.

**Steps**:

1. Check crash reporting for the error signature.
2. Identify the component tree causing the errors.
3. If caused by a data shape change from the API:
   - Check recent BFF or domain API deployments.
   - Add defensive checks or rollback the API change.
4. If caused by a client-side bug:
   - Prepare OTA hotfix.
5. If the error is recoverable (user can retry), lower severity.

### Analytics Data Quality (analytics-validation-low)

**Trigger**: Event validation rate < 99% over 1 hour.

**Steps**:

1. Check which events are failing validation.
2. Common causes:
   - New event type added without schema registration.
   - Client SDK version mismatch after deployment.
   - Event name format violation (not following ADR-0006).
3. If caused by new event type: register the schema.
4. If caused by SDK bug: prepare hotfix.
5. Invalid events are logged but not lost — they can be reprocessed.

---

## Escalation Paths

| Domain        | Primary Contact     | Backup Contact   | Slack Channel      |
| ------------- | ------------------- | ---------------- | ------------------ |
| Shell/Mobile  | Mobile on-call      | Platform lead    | #oshun-mobile      |
| Shell/Web     | Web on-call         | Platform lead    | #oshun-web         |
| BFF           | Backend on-call     | Platform lead    | #oshun-bff         |
| Tara          | Tara domain lead    | Platform on-call | #tara-incidents    |
| Veritas       | Veritas domain lead | Platform on-call | #veritas-incidents |
| Nyx           | Nyx domain lead     | Platform on-call | #nyx-incidents     |
| Arete         | Arete domain lead   | Platform on-call | #arete-incidents   |
| Auth/Identity | Auth team lead      | Platform lead    | #oshun-auth        |
| Analytics     | Data eng on-call    | Platform on-call | #oshun-analytics   |

---

## Communication Templates

### Incident Start

```
[P{severity}] {Alert Name}
Detected: {timestamp}
Impact: {description of user impact}
Investigating: {on-call name}
Status: Investigating
```

### Incident Update

```
[P{severity}] {Alert Name} - Update
Time: {timestamp}
Root cause: {identified/investigating}
Mitigation: {action taken or planned}
ETA: {estimated resolution time}
```

### Incident Resolution

```
[P{severity}] {Alert Name} - Resolved
Detected: {start time}
Resolved: {end time}
Duration: {total duration}
Root cause: {description}
Impact: {number of affected users/sessions}
Follow-up: {post-mortem scheduled Y/N}
```

---

## Post-Mortem Process

For any P0 or P1 incident:

1. Schedule post-mortem within 48 hours of resolution.
2. Document in `docs/incidents/YYYY-MM-DD-{slug}.md`:
   - Timeline of events
   - Root cause analysis (5 Whys)
   - Impact assessment
   - What went well
   - What could be improved
   - Action items with owners and deadlines
3. Share post-mortem in #oshun-engineering.
4. Track action items in the team's issue tracker.

---

## Dashboards

| Dashboard         | URL                                  | Purpose                    |
| ----------------- | ------------------------------------ | -------------------------- |
| BFF Overview      | `/admin/metrics/dashboard`           | Request volume and latency |
| BFF Prometheus    | `/admin/metrics/prometheus`          | Raw Prometheus metrics     |
| Crash Reporting   | BFF `/v1/mobile/crashes` logs        | Mobile crash events        |
| Web Errors        | BFF `/v1/web/errors` logs            | Web error events           |
| Web Vitals        | BFF `/v1/web/vitals` logs            | Core Web Vitals tracking   |
| Analytics Quality | Analytics validation report          | Event quality metrics      |
| Domain Health     | BFF dashboard `domainHealth` section | Domain API availability    |

---

## Useful Commands

```bash
# Check BFF health
curl -s http://localhost:4010/health | jq .

# Check BFF metrics
curl -s http://localhost:4010/admin/metrics/prometheus

# View recent crash reports
curl -s http://localhost:4010/v1/mobile/crashes/recent | jq .

# Check domain API health
curl -s http://localhost:4101/health  # Tara
curl -s http://localhost:4102/health  # Veritas
curl -s http://localhost:4103/health  # Nyx
curl -s http://localhost:4104/health  # Arete

# View BFF container logs
docker logs oshun-bff --tail 100 --follow

# Restart BFF
docker restart oshun-bff
```
