Incident response procedures and escalation paths for the OSHUN platform.
Severity Levels#
| Level | Definition | Response Time | Escalation |
|---|---|---|---|
| P0 | Service outage affecting all users (app crash, auth down) | 5 minutes | Immediate page |
| P1 | Major degradation affecting >10% of users | 15 minutes | Page on-call |
| P2 | Partial degradation, single domain or feature impacted | 1 hour | Slack alert |
| P3 | Minor issue, workaround available, no user-visible impact | 4 hours | Slack alert |
| P4 | Informational, no immediate action needed | Next business day | Slack channel |
Alert Response Procedures#
High Crash Rate (crash-rate-high)#
Trigger: Crash-free sessions rate < 99.5% over 15 minutes.
Steps:
- Check crash reporting dashboard for the top crash signature.
- Identify the affected release version(s) and platform(s).
- If crash is in the latest release:
- Check if a rollback OTA update is available.
- If OTA is not possible, prepare an emergency native build.
- If crash is in a domain module (Tara, Arete, Veritas, Nyx, Nisaba):
- Disable the domain module via feature flag if possible.
- Notify the domain team lead.
- Post status update in #oshun-incidents.
Fatal Crash Spike (fatal-crash-spike)#
Trigger: >10 fatal crashes in 5 minutes.
Steps:
- Immediately check if a new release was deployed in the last hour.
- If yes: initiate rollback (OTA or app store if needed).
- If no: check for backend dependency failures that could cause client crashes.
- Check device/OS distribution — may be platform-specific.
- Escalate to P0 if affecting >1% of daily active users.
High BFF Error Rate (bff-error-rate-high)#
Trigger: BFF 5xx error rate > 5% over 10 minutes.
Steps:
- Check BFF logs for the error pattern:bash
docker logs oshun-bff --tail 200 | grep "ERROR" - Check if the error is from a specific route or all routes.
- If route-specific: check the downstream domain API health.
- If global: check BFF process health (memory, CPU, event loop).
- If domain API is down:
- Enable graceful degradation (cached/fallback responses).
- Notify domain team.
- If BFF itself is failing:
- Check recent deployments.
- Restart BFF pods/containers.
- If persists, rollback to last known good deployment.
Critical BFF Latency (bff-latency-critical)#
Trigger: BFF P95 response time > 5 seconds over 10 minutes.
Steps:
- Check if latency is from a specific route or global.
- If route-specific:
- Check the downstream domain API response times.
- Check if caching is functioning (Redis health).
- If global:
- Check database connection pool health.
- Check for event loop blocking (long-running synchronous operations).
- Check memory usage for potential GC pressure.
- Temporary mitigations:
- Increase cache TTLs to reduce backend load.
- Enable request shedding for non-critical endpoints.
Slow Cold Start (startup-critical)#
Trigger: P95 cold start time > 8 seconds over 15 minutes.
Steps:
- Check if this coincides with a new release.
- Check JS bundle size — a large bundle increase causes cold start regression.
- Check if third-party SDKs were added or updated.
- If caused by new release: prepare hotfix to reduce bundle size.
- If not release-related: check if device distribution shifted (more low-end devices).
Domain Unavailable (domain-unavailable)#
Trigger: Domain API returns errors for 100% of requests over 5 minutes.
Steps:
- Identify which domain is affected.
- Check domain service health directly:bash
curl -s https://<domain-api>/health - Check if the issue is network/DNS or application-level.
- Enable fallback UI in the shell (offline cards, cached content).
- Notify domain team lead immediately.
- Post status update in #oshun-incidents with affected domain.
Error Boundary Spike (error-boundary-spike)#
Trigger: >50 error boundary triggers in 10 minutes.
Steps:
- Check crash reporting for the error signature.
- Identify the component tree causing the errors.
- If caused by a data shape change from the API:
- Check recent BFF or domain API deployments.
- Add defensive checks or rollback the API change.
- If caused by a client-side bug:
- Prepare OTA hotfix.
- If the error is recoverable (user can retry), lower severity.
Analytics Data Quality (analytics-validation-low)#
Trigger: Event validation rate < 99% over 1 hour.
Steps:
- Check which events are failing validation.
- Common causes:
- New event type added without schema registration.
- Client SDK version mismatch after deployment.
- Event name format violation (not following ADR-0006).
- If caused by new event type: register the schema.
- If caused by SDK bug: prepare hotfix.
- Invalid events are logged but not lost — they can be reprocessed.
Escalation Paths#
| Domain | Primary Contact | Backup Contact | Slack Channel |
|---|---|---|---|
| Shell/Mobile | Mobile on-call | Platform lead | #oshun-mobile |
| Shell/Web | Web on-call | Platform lead | #oshun-web |
| BFF | Backend on-call | Platform lead | #oshun-bff |
| Tara | Tara domain lead | Platform on-call | #tara-incidents |
| Veritas | Veritas domain lead | Platform on-call | #veritas-incidents |
| Nyx | Nyx domain lead | Platform on-call | #nyx-incidents |
| Arete | Arete domain lead | Platform on-call | #arete-incidents |
| Auth/Identity | Auth team lead | Platform lead | #oshun-auth |
| Analytics | Data eng on-call | Platform on-call | #oshun-analytics |
Communication Templates#
Incident Start#
text
[P{severity}] {Alert Name}
Detected: {timestamp}
Impact: {description of user impact}
Investigating: {on-call name}
Status: Investigating
Incident Update#
text
[P{severity}] {Alert Name} - Update
Time: {timestamp}
Root cause: {identified/investigating}
Mitigation: {action taken or planned}
ETA: {estimated resolution time}
Incident Resolution#
text
[P{severity}] {Alert Name} - Resolved
Detected: {start time}
Resolved: {end time}
Duration: {total duration}
Root cause: {description}
Impact: {number of affected users/sessions}
Follow-up: {post-mortem scheduled Y/N}
Post-Mortem Process#
For any P0 or P1 incident:
- Schedule post-mortem within 48 hours of resolution.
- Document in
docs/incidents/YYYY-MM-DD-{slug}.md:- Timeline of events
- Root cause analysis (5 Whys)
- Impact assessment
- What went well
- What could be improved
- Action items with owners and deadlines
- Share post-mortem in #oshun-engineering.
- Track action items in the team's issue tracker.
Dashboards#
| Dashboard | URL | Purpose |
|---|---|---|
| BFF Overview | /admin/metrics/dashboard |
Request volume and latency |
| BFF Prometheus | /admin/metrics/prometheus |
Raw Prometheus metrics |
| Crash Reporting | BFF /v1/mobile/crashes logs |
Mobile crash events |
| Web Errors | BFF /v1/web/errors logs |
Web error events |
| Web Vitals | BFF /v1/web/vitals logs |
Core Web Vitals tracking |
| Analytics Quality | Analytics validation report | Event quality metrics |
| Domain Health | BFF dashboard domainHealth section |
Domain API availability |
Useful Commands#
bash
# Check BFF health
curl -s http://localhost:4010/health | jq .
# Check BFF metrics
curl -s http://localhost:4010/admin/metrics/prometheus
# View recent crash reports
curl -s http://localhost:4010/v1/mobile/crashes/recent | jq .
# Check domain API health
curl -s http://localhost:4101/health # Tara
curl -s http://localhost:4102/health # Veritas
curl -s http://localhost:4103/health # Nyx
curl -s http://localhost:4104/health # Arete
# View BFF container logs
docker logs oshun-bff --tail 100 --follow
# Restart BFF
docker restart oshun-bff