Pre-Launch (T-7 days)#
Code Freeze and Final Build#
- Announce code freeze for OSHUN apps (mobile, web, BFF).
- Merge all approved PRs. No new feature work after freeze.
- Tag release:
git tag v1.0.0 && git push origin v1.0.0. - Build production binaries:
eas build --profile production --platform allpnpm nx run @oshun/web:buildpnpm nx run @oshun/bff:build
- Run full quality gate suite:
pnpm nx run-many --target=lint --projects=@oshun/*pnpm nx run-many --target=typecheck --projects=@oshun/*pnpm nx run-many --target=test --projects=@oshun/*pnpm nx run @oshun/web:lighthouse
- Verify all quality gates pass with zero errors.
Store Submissions#
- iOS: Submit to App Store Connect for review.
- Upload via
eas submit --profile production --platform ios. - Fill in release notes, screenshots, and metadata from
store/metadata.json. - Select phased release (7-day).
- Ensure demo account is active.
- Upload via
- Android: Submit to Google Play Console.
- Upload via
eas submit --profile production --platform android. - Fill in store listing from
store/metadata.json. - Set staged rollout to 1%.
- Ensure data safety section is current.
- Upload via
- Web: Deploy to production via standard workflow.
- Trigger:
gh workflow run oshun-web-deploy.yml -f environment=production -f strategy=standard - Verify health check passes.
- Trigger:
Infrastructure Readiness#
- BFF scaled to launch capacity (minimum 3 replicas).
- CDN cache warmed for static assets.
- Redis cache preheated with popular domain content.
- Feature flags set to launch configuration:
domainSwitcher: truequickActionsTray: trueuniversalSearch: trueofflineMode: truehomeLayoutV2: false(save for post-launch experiment)
- Sentry release version tagged.
- Analytics write key for production configured.
Monitoring and Alerts#
- Verify all alert rules are active (16 rules per
OSHUN_ALERT_RULES). - Verify PagerDuty escalation policies for P0/P1.
- Verify Sentry alert thresholds:
- Crash rate > 0.5% → P1 alert
- Error rate > 5% → P1 alert
- Latency P95 > 3s → P2 alert
- Verify BFF metrics dashboard shows all endpoints.
- Verify domain health probes return 200 for all five customer domains.
- Verify critical shared grounding dependencies are healthy for grounded flows.
- Set up launch-specific Slack channel:
#oshun-launch-war-room.
Communications#
- Draft launch announcement for internal team.
- Draft app store release notes.
- Draft social media posts (if applicable).
- Prepare known-issues page at
oshun.app/known-issues. - Notify domain teams (Tara, Arete, Veritas, Nyx, Nisaba) of launch timeline.
Launch Day (T-0)#
Timeline#
| Time | Action | Owner |
|---|---|---|
| 09:00 | War room opens. All hands in #oshun-launch-war-room. |
On-call |
| 09:15 | Final health check: BFF, domains, web, CDN. | Backend |
| 09:30 | iOS: Release approved build (phased release Day 1 = 1%). | Mobile |
| 09:35 | Android: Begin staged rollout at 1%. | Mobile |
| 09:40 | Web: Confirm production deployment is live. | Frontend |
| 09:45 | Send internal launch announcement. | Product |
| 10:00 | Monitor real-time dashboards (15-minute cadence). | On-call |
| 12:00 | First checkpoint: crash rate, error rate, latency. | On-call |
| 15:00 | Second checkpoint: user signups, onboarding completion. | Product |
| 18:00 | End-of-day review: metrics summary, open issues. | All |
| 18:30 | War room closes for Day 1 (on-call remains active). | On-call |
Real-Time Dashboard Monitoring#
Monitor these dashboards continuously during launch window (09:00-18:00):
-
Crash Rate Dashboard
- iOS crash rate (< 0.5%)
- Android crash rate (< 0.5%)
- ANR rate (< 0.2%)
-
BFF Health Dashboard
- Request rate (expected: 10-100 RPS initially)
- Error rate (< 1%)
- P95 latency (< 500ms)
- Per-route breakdown
-
Domain Availability Dashboard
- Tara API health
- Arete API health
- Veritas API health
- Nyx API health
- Nisaba API health
- Response times per domain
-
Grounding Dependency Dashboard
- Sophia API health
- retrieval/citation pipeline health
- evidence-service latency
- grounded-response error rate
-
User Funnel Dashboard
- App installs / web visits
- Onboarding started → completed rate
- First domain launch rate
- Day-0 retention signal
-
Infrastructure Dashboard
- BFF CPU/memory utilization
- Redis hit rate and memory
- CDN cache hit ratio
- SSL certificate validity
Incident Triage Protocol#
During launch window, incidents follow an accelerated triage:
| Severity | Response Time | Escalation | Action |
|---|---|---|---|
| P0 | 5 min | Page on-call + engineering lead | Halt rollout, OTA fix or rollback |
| P1 | 15 min | Alert on-call + war room | Pause rollout, assess fix-forward vs. rollback |
| P2 | 30 min | Post in war room | Continue rollout, fix in next OTA |
| P3 | 1 hour | Log for post-launch | Track, no immediate action |
Rollout pause triggers (automatic):
- Crash rate exceeds 1% for any 5-minute window.
- BFF error rate exceeds 5% for any 5-minute window.
- Any domain returns 5xx for > 2 consecutive health checks.
- User-reported P0 issue confirmed by on-call.
Rollout resume criteria:
- Root cause identified and fixed (or mitigated via feature flag).
- Crash rate returns below 0.5% for 30 consecutive minutes.
- On-call and engineering lead both approve.
Post-Launch Day 1-3#
Daily Standup (09:30, 15 min)#
Agenda:
- Metrics review: crashes, errors, latency, signups, retention.
- Open issues: P0/P1 count, new reports, in-progress fixes.
- Rollout progress: current percentage, next stage ETA.
- Decision: advance rollout, hold, or roll back.
Android Staged Rollout Progression#
| Day | Target % | Gate Criteria |
|---|---|---|
| 0 | 1% | Launch day — observe 24 hours |
| 1 | 5% | Crash rate < 0.5%, zero P0/P1 |
| 2 | 25% | D1 retention > 35%, ANR rate < 0.2% |
| 3 | 50% | No new P0/P1, user rating >= 3.5 |
| 5 | 100% | Stable for 48 hours at 50% |
iOS Phased Release#
iOS App Store Connect manages phased release automatically (1% → 2% → 5% → 10% → 20% → 50% → 100% over 7 days). Pause if:
- Crash rate exceeds threshold.
- Critical review feedback from App Review.
Known Issues Page#
Publish at oshun.app/known-issues. Update within 30 minutes of any confirmed
user-facing issue.
Template:
markdown
# OSHUN Known Issues
Last updated: [timestamp]
## Active Issues
### [Issue Title]
- **Status:** Investigating / Fix in progress / Fixed (pending rollout)
- **Affected:** iOS / Android / Web / All
- **Description:** Brief user-facing description.
- **Workaround:** Steps to avoid the issue, if any.
- **ETA:** Expected fix timeline.
## Recently Resolved
### [Issue Title]
- **Resolved:** [date]
- **Fix:** Included in version [X.Y.Z] / OTA update [ID].
Communication Templates#
Incident Start#
text
🚨 [P0/P1] OSHUN Incident — [Brief Title]
Impact: [What users are experiencing]
Affected: [iOS/Android/Web, percentage of users]
Status: Investigating
On-call: [@name]
War room: #oshun-launch-war-room
Next update in 15 minutes.
Incident Update#
text
🔄 OSHUN Incident Update — [Brief Title]
Root cause: [Identified/Unknown]
Mitigation: [Actions taken]
Status: [Mitigating / Fix deployed / Monitoring]
Rollout: [Paused at X% / Resumed]
Next update in [30 min / 1 hour].
Incident Resolution#
text
✅ OSHUN Incident Resolved — [Brief Title]
Duration: [X hours Y minutes]
Root cause: [Brief explanation]
Fix: [What was deployed]
Users affected: [Approximate count]
Rollout: [Resumed / Already at 100%]
Post-mortem scheduled for [date].
Emergency Contacts#
| Role | Primary | Backup |
|---|---|---|
| Engineering On-Call | [Rotation] | [Rotation] |
| Engineering Lead | [Name] | [Name] |
| Product Lead | [Name] | [Name] |
| Infrastructure | [Name] | [Name] |
| Domain: Tara | [Name] | [Name] |
| Domain: Veritas | [Name] | [Name] |
| Domain: Nyx | [Name] | [Name] |
| Domain: Arete | [Name] | [Name] |