Eve library

OSHUN On-Call Runbook

Runbook

Revision 1

On this page

Incident response procedures and escalation paths for the OSHUN platform.


Severity Levels#

Level Definition Response Time Escalation
P0 Service outage affecting all users (app crash, auth down) 5 minutes Immediate page
P1 Major degradation affecting >10% of users 15 minutes Page on-call
P2 Partial degradation, single domain or feature impacted 1 hour Slack alert
P3 Minor issue, workaround available, no user-visible impact 4 hours Slack alert
P4 Informational, no immediate action needed Next business day Slack channel

Alert Response Procedures#

High Crash Rate (crash-rate-high)#

Trigger: Crash-free sessions rate < 99.5% over 15 minutes.

Steps:

  1. Check crash reporting dashboard for the top crash signature.
  2. Identify the affected release version(s) and platform(s).
  3. If crash is in the latest release:
    • Check if a rollback OTA update is available.
    • If OTA is not possible, prepare an emergency native build.
  4. If crash is in a domain module (Tara, Arete, Veritas, Nyx, Nisaba):
    • Disable the domain module via feature flag if possible.
    • Notify the domain team lead.
  5. Post status update in #oshun-incidents.

Fatal Crash Spike (fatal-crash-spike)#

Trigger: >10 fatal crashes in 5 minutes.

Steps:

  1. Immediately check if a new release was deployed in the last hour.
  2. If yes: initiate rollback (OTA or app store if needed).
  3. If no: check for backend dependency failures that could cause client crashes.
  4. Check device/OS distribution — may be platform-specific.
  5. Escalate to P0 if affecting >1% of daily active users.

High BFF Error Rate (bff-error-rate-high)#

Trigger: BFF 5xx error rate > 5% over 10 minutes.

Steps:

  1. Check BFF logs for the error pattern:
    bash
    docker logs oshun-bff --tail 200 | grep "ERROR"
    
  2. Check if the error is from a specific route or all routes.
  3. If route-specific: check the downstream domain API health.
  4. If global: check BFF process health (memory, CPU, event loop).
  5. If domain API is down:
    • Enable graceful degradation (cached/fallback responses).
    • Notify domain team.
  6. If BFF itself is failing:
    • Check recent deployments.
    • Restart BFF pods/containers.
    • If persists, rollback to last known good deployment.

Critical BFF Latency (bff-latency-critical)#

Trigger: BFF P95 response time > 5 seconds over 10 minutes.

Steps:

  1. Check if latency is from a specific route or global.
  2. If route-specific:
    • Check the downstream domain API response times.
    • Check if caching is functioning (Redis health).
  3. If global:
    • Check database connection pool health.
    • Check for event loop blocking (long-running synchronous operations).
    • Check memory usage for potential GC pressure.
  4. Temporary mitigations:
    • Increase cache TTLs to reduce backend load.
    • Enable request shedding for non-critical endpoints.

Slow Cold Start (startup-critical)#

Trigger: P95 cold start time > 8 seconds over 15 minutes.

Steps:

  1. Check if this coincides with a new release.
  2. Check JS bundle size — a large bundle increase causes cold start regression.
  3. Check if third-party SDKs were added or updated.
  4. If caused by new release: prepare hotfix to reduce bundle size.
  5. If not release-related: check if device distribution shifted (more low-end devices).

Domain Unavailable (domain-unavailable)#

Trigger: Domain API returns errors for 100% of requests over 5 minutes.

Steps:

  1. Identify which domain is affected.
  2. Check domain service health directly:
    bash
    curl -s https://<domain-api>/health
    
  3. Check if the issue is network/DNS or application-level.
  4. Enable fallback UI in the shell (offline cards, cached content).
  5. Notify domain team lead immediately.
  6. Post status update in #oshun-incidents with affected domain.

Error Boundary Spike (error-boundary-spike)#

Trigger: >50 error boundary triggers in 10 minutes.

Steps:

  1. Check crash reporting for the error signature.
  2. Identify the component tree causing the errors.
  3. If caused by a data shape change from the API:
    • Check recent BFF or domain API deployments.
    • Add defensive checks or rollback the API change.
  4. If caused by a client-side bug:
    • Prepare OTA hotfix.
  5. If the error is recoverable (user can retry), lower severity.

Analytics Data Quality (analytics-validation-low)#

Trigger: Event validation rate < 99% over 1 hour.

Steps:

  1. Check which events are failing validation.
  2. Common causes:
    • New event type added without schema registration.
    • Client SDK version mismatch after deployment.
    • Event name format violation (not following ADR-0006).
  3. If caused by new event type: register the schema.
  4. If caused by SDK bug: prepare hotfix.
  5. Invalid events are logged but not lost — they can be reprocessed.

Escalation Paths#

Domain Primary Contact Backup Contact Slack Channel
Shell/Mobile Mobile on-call Platform lead #oshun-mobile
Shell/Web Web on-call Platform lead #oshun-web
BFF Backend on-call Platform lead #oshun-bff
Tara Tara domain lead Platform on-call #tara-incidents
Veritas Veritas domain lead Platform on-call #veritas-incidents
Nyx Nyx domain lead Platform on-call #nyx-incidents
Arete Arete domain lead Platform on-call #arete-incidents
Auth/Identity Auth team lead Platform lead #oshun-auth
Analytics Data eng on-call Platform on-call #oshun-analytics

Communication Templates#

Incident Start#

text
[P{severity}] {Alert Name}
Detected: {timestamp}
Impact: {description of user impact}
Investigating: {on-call name}
Status: Investigating

Incident Update#

text
[P{severity}] {Alert Name} - Update
Time: {timestamp}
Root cause: {identified/investigating}
Mitigation: {action taken or planned}
ETA: {estimated resolution time}

Incident Resolution#

text
[P{severity}] {Alert Name} - Resolved
Detected: {start time}
Resolved: {end time}
Duration: {total duration}
Root cause: {description}
Impact: {number of affected users/sessions}
Follow-up: {post-mortem scheduled Y/N}

Post-Mortem Process#

For any P0 or P1 incident:

  1. Schedule post-mortem within 48 hours of resolution.
  2. Document in docs/incidents/YYYY-MM-DD-{slug}.md:
    • Timeline of events
    • Root cause analysis (5 Whys)
    • Impact assessment
    • What went well
    • What could be improved
    • Action items with owners and deadlines
  3. Share post-mortem in #oshun-engineering.
  4. Track action items in the team's issue tracker.

Dashboards#

Dashboard URL Purpose
BFF Overview /admin/metrics/dashboard Request volume and latency
BFF Prometheus /admin/metrics/prometheus Raw Prometheus metrics
Crash Reporting BFF /v1/mobile/crashes logs Mobile crash events
Web Errors BFF /v1/web/errors logs Web error events
Web Vitals BFF /v1/web/vitals logs Core Web Vitals tracking
Analytics Quality Analytics validation report Event quality metrics
Domain Health BFF dashboard domainHealth section Domain API availability

Useful Commands#

bash
# Check BFF health
curl -s http://localhost:4010/health | jq .

# Check BFF metrics
curl -s http://localhost:4010/admin/metrics/prometheus

# View recent crash reports
curl -s http://localhost:4010/v1/mobile/crashes/recent | jq .

# Check domain API health
curl -s http://localhost:4101/health  # Tara
curl -s http://localhost:4102/health  # Veritas
curl -s http://localhost:4103/health  # Nyx
curl -s http://localhost:4104/health  # Arete

# View BFF container logs
docker logs oshun-bff --tail 100 --follow

# Restart BFF
docker restart oshun-bff