Eve library

OSHUN Launch Runbook

Runbook

Revision 1

On this page

Pre-Launch (T-7 days)#

Code Freeze and Final Build#

  • Announce code freeze for OSHUN apps (mobile, web, BFF).
  • Merge all approved PRs. No new feature work after freeze.
  • Tag release: git tag v1.0.0 && git push origin v1.0.0.
  • Build production binaries:
    • eas build --profile production --platform all
    • pnpm nx run @oshun/web:build
    • pnpm nx run @oshun/bff:build
  • Run full quality gate suite:
    • pnpm nx run-many --target=lint --projects=@oshun/*
    • pnpm nx run-many --target=typecheck --projects=@oshun/*
    • pnpm nx run-many --target=test --projects=@oshun/*
    • pnpm nx run @oshun/web:lighthouse
  • Verify all quality gates pass with zero errors.

Store Submissions#

  • iOS: Submit to App Store Connect for review.
    • Upload via eas submit --profile production --platform ios.
    • Fill in release notes, screenshots, and metadata from store/metadata.json.
    • Select phased release (7-day).
    • Ensure demo account is active.
  • Android: Submit to Google Play Console.
    • Upload via eas submit --profile production --platform android.
    • Fill in store listing from store/metadata.json.
    • Set staged rollout to 1%.
    • Ensure data safety section is current.
  • Web: Deploy to production via standard workflow.
    • Trigger: gh workflow run oshun-web-deploy.yml -f environment=production -f strategy=standard
    • Verify health check passes.

Infrastructure Readiness#

  • BFF scaled to launch capacity (minimum 3 replicas).
  • CDN cache warmed for static assets.
  • Redis cache preheated with popular domain content.
  • Feature flags set to launch configuration:
    • domainSwitcher: true
    • quickActionsTray: true
    • universalSearch: true
    • offlineMode: true
    • homeLayoutV2: false (save for post-launch experiment)
  • Sentry release version tagged.
  • Analytics write key for production configured.

Monitoring and Alerts#

  • Verify all alert rules are active (16 rules per OSHUN_ALERT_RULES).
  • Verify PagerDuty escalation policies for P0/P1.
  • Verify Sentry alert thresholds:
    • Crash rate > 0.5% → P1 alert
    • Error rate > 5% → P1 alert
    • Latency P95 > 3s → P2 alert
  • Verify BFF metrics dashboard shows all endpoints.
  • Verify domain health probes return 200 for all five customer domains.
  • Verify critical shared grounding dependencies are healthy for grounded flows.
  • Set up launch-specific Slack channel: #oshun-launch-war-room.

Communications#

  • Draft launch announcement for internal team.
  • Draft app store release notes.
  • Draft social media posts (if applicable).
  • Prepare known-issues page at oshun.app/known-issues.
  • Notify domain teams (Tara, Arete, Veritas, Nyx, Nisaba) of launch timeline.

Launch Day (T-0)#

Timeline#

Time Action Owner
09:00 War room opens. All hands in #oshun-launch-war-room. On-call
09:15 Final health check: BFF, domains, web, CDN. Backend
09:30 iOS: Release approved build (phased release Day 1 = 1%). Mobile
09:35 Android: Begin staged rollout at 1%. Mobile
09:40 Web: Confirm production deployment is live. Frontend
09:45 Send internal launch announcement. Product
10:00 Monitor real-time dashboards (15-minute cadence). On-call
12:00 First checkpoint: crash rate, error rate, latency. On-call
15:00 Second checkpoint: user signups, onboarding completion. Product
18:00 End-of-day review: metrics summary, open issues. All
18:30 War room closes for Day 1 (on-call remains active). On-call

Real-Time Dashboard Monitoring#

Monitor these dashboards continuously during launch window (09:00-18:00):

  1. Crash Rate Dashboard

    • iOS crash rate (< 0.5%)
    • Android crash rate (< 0.5%)
    • ANR rate (< 0.2%)
  2. BFF Health Dashboard

    • Request rate (expected: 10-100 RPS initially)
    • Error rate (< 1%)
    • P95 latency (< 500ms)
    • Per-route breakdown
  3. Domain Availability Dashboard

    • Tara API health
    • Arete API health
    • Veritas API health
    • Nyx API health
    • Nisaba API health
    • Response times per domain
  4. Grounding Dependency Dashboard

    • Sophia API health
    • retrieval/citation pipeline health
    • evidence-service latency
    • grounded-response error rate
  5. User Funnel Dashboard

    • App installs / web visits
    • Onboarding started → completed rate
    • First domain launch rate
    • Day-0 retention signal
  6. Infrastructure Dashboard

    • BFF CPU/memory utilization
    • Redis hit rate and memory
    • CDN cache hit ratio
    • SSL certificate validity

Incident Triage Protocol#

During launch window, incidents follow an accelerated triage:

Severity Response Time Escalation Action
P0 5 min Page on-call + engineering lead Halt rollout, OTA fix or rollback
P1 15 min Alert on-call + war room Pause rollout, assess fix-forward vs. rollback
P2 30 min Post in war room Continue rollout, fix in next OTA
P3 1 hour Log for post-launch Track, no immediate action

Rollout pause triggers (automatic):

  • Crash rate exceeds 1% for any 5-minute window.
  • BFF error rate exceeds 5% for any 5-minute window.
  • Any domain returns 5xx for > 2 consecutive health checks.
  • User-reported P0 issue confirmed by on-call.

Rollout resume criteria:

  • Root cause identified and fixed (or mitigated via feature flag).
  • Crash rate returns below 0.5% for 30 consecutive minutes.
  • On-call and engineering lead both approve.

Post-Launch Day 1-3#

Daily Standup (09:30, 15 min)#

Agenda:

  1. Metrics review: crashes, errors, latency, signups, retention.
  2. Open issues: P0/P1 count, new reports, in-progress fixes.
  3. Rollout progress: current percentage, next stage ETA.
  4. Decision: advance rollout, hold, or roll back.

Android Staged Rollout Progression#

Day Target % Gate Criteria
0 1% Launch day — observe 24 hours
1 5% Crash rate < 0.5%, zero P0/P1
2 25% D1 retention > 35%, ANR rate < 0.2%
3 50% No new P0/P1, user rating >= 3.5
5 100% Stable for 48 hours at 50%

iOS Phased Release#

iOS App Store Connect manages phased release automatically (1% → 2% → 5% → 10% → 20% → 50% → 100% over 7 days). Pause if:

  • Crash rate exceeds threshold.
  • Critical review feedback from App Review.

Known Issues Page#

Publish at oshun.app/known-issues. Update within 30 minutes of any confirmed user-facing issue.

Template:

markdown
# OSHUN Known Issues

Last updated: [timestamp]

## Active Issues

### [Issue Title]

- **Status:** Investigating / Fix in progress / Fixed (pending rollout)
- **Affected:** iOS / Android / Web / All
- **Description:** Brief user-facing description.
- **Workaround:** Steps to avoid the issue, if any.
- **ETA:** Expected fix timeline.

## Recently Resolved

### [Issue Title]

- **Resolved:** [date]
- **Fix:** Included in version [X.Y.Z] / OTA update [ID].

Communication Templates#

Incident Start#

text
🚨 [P0/P1] OSHUN Incident — [Brief Title]

Impact: [What users are experiencing]
Affected: [iOS/Android/Web, percentage of users]
Status: Investigating
On-call: [@name]
War room: #oshun-launch-war-room

Next update in 15 minutes.

Incident Update#

text
🔄 OSHUN Incident Update — [Brief Title]

Root cause: [Identified/Unknown]
Mitigation: [Actions taken]
Status: [Mitigating / Fix deployed / Monitoring]
Rollout: [Paused at X% / Resumed]

Next update in [30 min / 1 hour].

Incident Resolution#

text
✅ OSHUN Incident Resolved — [Brief Title]

Duration: [X hours Y minutes]
Root cause: [Brief explanation]
Fix: [What was deployed]
Users affected: [Approximate count]
Rollout: [Resumed / Already at 100%]

Post-mortem scheduled for [date].

Emergency Contacts#

Role Primary Backup
Engineering On-Call [Rotation] [Rotation]
Engineering Lead [Name] [Name]
Product Lead [Name] [Name]
Infrastructure [Name] [Name]
Domain: Tara [Name] [Name]
Domain: Veritas [Name] [Name]
Domain: Nyx [Name] [Name]
Domain: Arete [Name] [Name]