# OSHUN Launch Runbook

## Pre-Launch (T-7 days)

### Code Freeze and Final Build

- [ ] Announce code freeze for OSHUN apps (mobile, web, BFF).
- [ ] Merge all approved PRs. No new feature work after freeze.
- [ ] Tag release: `git tag v1.0.0 && git push origin v1.0.0`.
- [ ] Build production binaries:
  - `eas build --profile production --platform all`
  - `pnpm nx run @oshun/web:build`
  - `pnpm nx run @oshun/bff:build`
- [ ] Run full quality gate suite:
  - `pnpm nx run-many --target=lint --projects=@oshun/*`
  - `pnpm nx run-many --target=typecheck --projects=@oshun/*`
  - `pnpm nx run-many --target=test --projects=@oshun/*`
  - `pnpm nx run @oshun/web:lighthouse`
- [ ] Verify all quality gates pass with zero errors.

### Store Submissions

- [ ] iOS: Submit to App Store Connect for review.
  - Upload via `eas submit --profile production --platform ios`.
  - Fill in release notes, screenshots, and metadata from `store/metadata.json`.
  - Select phased release (7-day).
  - Ensure demo account is active.
- [ ] Android: Submit to Google Play Console.
  - Upload via `eas submit --profile production --platform android`.
  - Fill in store listing from `store/metadata.json`.
  - Set staged rollout to 1%.
  - Ensure data safety section is current.
- [ ] Web: Deploy to production via standard workflow.
  - Trigger:
    `gh workflow run oshun-web-deploy.yml -f environment=production -f strategy=standard`
  - Verify health check passes.

### Infrastructure Readiness

- [ ] BFF scaled to launch capacity (minimum 3 replicas).
- [ ] CDN cache warmed for static assets.
- [ ] Redis cache preheated with popular domain content.
- [ ] Feature flags set to launch configuration:
  - `domainSwitcher: true`
  - `quickActionsTray: true`
  - `universalSearch: true`
  - `offlineMode: true`
  - `homeLayoutV2: false` (save for post-launch experiment)
- [ ] Sentry release version tagged.
- [ ] Analytics write key for production configured.

### Monitoring and Alerts

- [ ] Verify all alert rules are active (16 rules per `OSHUN_ALERT_RULES`).
- [ ] Verify PagerDuty escalation policies for P0/P1.
- [ ] Verify Sentry alert thresholds:
  - Crash rate > 0.5% → P1 alert
  - Error rate > 5% → P1 alert
  - Latency P95 > 3s → P2 alert
- [ ] Verify BFF metrics dashboard shows all endpoints.
- [ ] Verify domain health probes return 200 for all five customer domains.
- [ ] Verify critical shared grounding dependencies are healthy for grounded
      flows.
- [ ] Set up launch-specific Slack channel: `#oshun-launch-war-room`.

### Communications

- [ ] Draft launch announcement for internal team.
- [ ] Draft app store release notes.
- [ ] Draft social media posts (if applicable).
- [ ] Prepare known-issues page at `oshun.app/known-issues`.
- [ ] Notify domain teams (Tara, Arete, Veritas, Nyx, Nisaba) of launch
      timeline.

---

## Launch Day (T-0)

### Timeline

| Time  | Action                                                   | Owner    |
| ----- | -------------------------------------------------------- | -------- |
| 09:00 | War room opens. All hands in `#oshun-launch-war-room`.   | On-call  |
| 09:15 | Final health check: BFF, domains, web, CDN.              | Backend  |
| 09:30 | iOS: Release approved build (phased release Day 1 = 1%). | Mobile   |
| 09:35 | Android: Begin staged rollout at 1%.                     | Mobile   |
| 09:40 | Web: Confirm production deployment is live.              | Frontend |
| 09:45 | Send internal launch announcement.                       | Product  |
| 10:00 | Monitor real-time dashboards (15-minute cadence).        | On-call  |
| 12:00 | First checkpoint: crash rate, error rate, latency.       | On-call  |
| 15:00 | Second checkpoint: user signups, onboarding completion.  | Product  |
| 18:00 | End-of-day review: metrics summary, open issues.         | All      |
| 18:30 | War room closes for Day 1 (on-call remains active).      | On-call  |

### Real-Time Dashboard Monitoring

Monitor these dashboards continuously during launch window (09:00-18:00):

1. **Crash Rate Dashboard**
   - iOS crash rate (< 0.5%)
   - Android crash rate (< 0.5%)
   - ANR rate (< 0.2%)

2. **BFF Health Dashboard**
   - Request rate (expected: 10-100 RPS initially)
   - Error rate (< 1%)
   - P95 latency (< 500ms)
   - Per-route breakdown

3. **Domain Availability Dashboard**
   - Tara API health
   - Arete API health
   - Veritas API health
   - Nyx API health
   - Nisaba API health
   - Response times per domain

4. **Grounding Dependency Dashboard**
   - Sophia API health
   - retrieval/citation pipeline health
   - evidence-service latency
   - grounded-response error rate

5. **User Funnel Dashboard**
   - App installs / web visits
   - Onboarding started → completed rate
   - First domain launch rate
   - Day-0 retention signal

6. **Infrastructure Dashboard**
   - BFF CPU/memory utilization
   - Redis hit rate and memory
   - CDN cache hit ratio
   - SSL certificate validity

### Incident Triage Protocol

During launch window, incidents follow an accelerated triage:

| Severity | Response Time | Escalation                      | Action                                         |
| -------- | ------------- | ------------------------------- | ---------------------------------------------- |
| P0       | 5 min         | Page on-call + engineering lead | Halt rollout, OTA fix or rollback              |
| P1       | 15 min        | Alert on-call + war room        | Pause rollout, assess fix-forward vs. rollback |
| P2       | 30 min        | Post in war room                | Continue rollout, fix in next OTA              |
| P3       | 1 hour        | Log for post-launch             | Track, no immediate action                     |

**Rollout pause triggers (automatic):**

- Crash rate exceeds 1% for any 5-minute window.
- BFF error rate exceeds 5% for any 5-minute window.
- Any domain returns 5xx for > 2 consecutive health checks.
- User-reported P0 issue confirmed by on-call.

**Rollout resume criteria:**

- Root cause identified and fixed (or mitigated via feature flag).
- Crash rate returns below 0.5% for 30 consecutive minutes.
- On-call and engineering lead both approve.

---

## Post-Launch Day 1-3

### Daily Standup (09:30, 15 min)

Agenda:

1. Metrics review: crashes, errors, latency, signups, retention.
2. Open issues: P0/P1 count, new reports, in-progress fixes.
3. Rollout progress: current percentage, next stage ETA.
4. Decision: advance rollout, hold, or roll back.

### Android Staged Rollout Progression

| Day | Target % | Gate Criteria                       |
| --- | -------- | ----------------------------------- |
| 0   | 1%       | Launch day — observe 24 hours       |
| 1   | 5%       | Crash rate < 0.5%, zero P0/P1       |
| 2   | 25%      | D1 retention > 35%, ANR rate < 0.2% |
| 3   | 50%      | No new P0/P1, user rating >= 3.5    |
| 5   | 100%     | Stable for 48 hours at 50%          |

### iOS Phased Release

iOS App Store Connect manages phased release automatically (1% → 2% → 5% → 10% →
20% → 50% → 100% over 7 days). Pause if:

- Crash rate exceeds threshold.
- Critical review feedback from App Review.

---

## Known Issues Page

Publish at `oshun.app/known-issues`. Update within 30 minutes of any confirmed
user-facing issue.

Template:

```markdown
# OSHUN Known Issues

Last updated: [timestamp]

## Active Issues

### [Issue Title]

- **Status:** Investigating / Fix in progress / Fixed (pending rollout)
- **Affected:** iOS / Android / Web / All
- **Description:** Brief user-facing description.
- **Workaround:** Steps to avoid the issue, if any.
- **ETA:** Expected fix timeline.

## Recently Resolved

### [Issue Title]

- **Resolved:** [date]
- **Fix:** Included in version [X.Y.Z] / OTA update [ID].
```

---

## Communication Templates

### Incident Start

```
🚨 [P0/P1] OSHUN Incident — [Brief Title]

Impact: [What users are experiencing]
Affected: [iOS/Android/Web, percentage of users]
Status: Investigating
On-call: [@name]
War room: #oshun-launch-war-room

Next update in 15 minutes.
```

### Incident Update

```
🔄 OSHUN Incident Update — [Brief Title]

Root cause: [Identified/Unknown]
Mitigation: [Actions taken]
Status: [Mitigating / Fix deployed / Monitoring]
Rollout: [Paused at X% / Resumed]

Next update in [30 min / 1 hour].
```

### Incident Resolution

```
✅ OSHUN Incident Resolved — [Brief Title]

Duration: [X hours Y minutes]
Root cause: [Brief explanation]
Fix: [What was deployed]
Users affected: [Approximate count]
Rollout: [Resumed / Already at 100%]

Post-mortem scheduled for [date].
```

---

## Emergency Contacts

| Role                | Primary    | Backup     |
| ------------------- | ---------- | ---------- |
| Engineering On-Call | [Rotation] | [Rotation] |
| Engineering Lead    | [Name]     | [Name]     |
| Product Lead        | [Name]     | [Name]     |
| Infrastructure      | [Name]     | [Name]     |
| Domain: Tara        | [Name]     | [Name]     |
| Domain: Veritas     | [Name]     | [Name]     |
| Domain: Nyx         | [Name]     | [Name]     |
| Domain: Arete       | [Name]     | [Name]     |
