Eve library

Escalation Matrix

Runbook

Revision 1

On this page

Purpose: Clear guidelines for when and how to escalate incidents during on-call shifts

Quick Reference Escalation Decision Tree#

text
Incident Detected
       ↓
   Assess Severity
       ↓
    ┌──────┐     ┌──────┐     ┌──────┐     ┌──────┐
    │  P0  │     │  P1  │     │  P2  │     │  P3  │
    └──┬───┘     └──┬───┘     └──┬───┘     └──┬───┘
       ↓            ↓            ↓            ↓
  Immediate      30 mins      2 hours    Next Bus Day
  Escalation    Response      Response     Response
       ↓            ↓            ↓            ↓
  IC + Platform   Platform    Team Lead      None
     Lead          Lead

Severity Definitions and Escalation Triggers#

P0 - Critical (Immediate Escalation)#

Criteria#

  • Platform Down: >50% of services unavailable
  • Data Loss: Confirmed data corruption or loss
  • Security Breach: Unauthorized access detected
  • Customer Data Exposure: PII/sensitive data leaked
  • Revenue Impact: Payment processing completely down
  • Core Function Failure: Authentication system down

Escalation Triggers#

  • Immediate (0 minutes): Start incident response
  • 5 minutes: Engage Incident Commander
  • 15 minutes: Escalate to Platform Lead
  • 30 minutes: Escalate to Engineering Director
  • 60 minutes: Executive notification

Response Team Formation#

text
Primary: On-Call Engineer
Secondary: Incident Commander
Tertiary: Platform Lead
Quaternary: Engineering Director

P1 - High (Timed Escalation)#

Criteria#

  • Service Degradation: 20-50% error rates
  • Performance Issues: >5s response times
  • Partial Outage: Single critical service down with workarounds
  • User Impact: >10% of users affected
  • Integration Failures: Key third-party service failures

Escalation Triggers#

  • 30 minutes: If no significant progress
  • 1 hour: Escalate to Platform Lead
  • 2 hours: Incident Commander if needed
  • 4 hours: Engineering Director notification

Response Team Formation#

text
Primary: On-Call Engineer
Secondary: Platform Lead (after 1 hour)
Tertiary: Subject Matter Expert

P2 - Medium (Business Hours Escalation)#

Criteria#

  • Minor Degradation: 5-20% error rates
  • Single Component Failure: With functional redundancy
  • Non-Critical Issues: Limited user impact
  • Monitoring Alerts: Threshold breaches without immediate impact

Escalation Triggers#

  • 2 hours: If during business hours
  • Next business day: If after hours
  • 4 hours: Team Lead notification

Response Team Formation#

text
Primary: On-Call Engineer
Secondary: Team Lead (business hours only)

P3 - Low (No Immediate Escalation)#

Criteria#

  • Capacity Warnings: Resource usage alerts
  • Monitoring Issues: Dashboard/alerting problems
  • Maintenance Notifications: Planned work notifications
  • Documentation Issues: Runbook or process gaps

Escalation Triggers#

  • Next business day: Normal business process
  • Never urgent: Can wait for business hours

Response Team Formation#

text
Primary: On-Call Engineer (acknowledge only)
Secondary: Team Lead (next business day)

Escalation Contact Matrix#

Primary Contacts#

Role Primary Contact Backup Contact Response SLA
Incident Commander @incident-commander @ic-backup 5 minutes
Platform Lead @platform-lead @platform-deputy 15 minutes
Security Lead @security-lead @security-deputy 10 minutes
Engineering Director @engineering-director @eng-director-backup 30 minutes
Database Expert @dba-lead @dba-backup 20 minutes
Infrastructure Expert @infra-lead @infra-backup 15 minutes
AI/ML Expert @ai-lead @ai-backup 30 minutes

Contact Methods#

Immediate Response Required (P0)#

  1. PagerDuty: Page directly
  2. Phone Call: Use emergency contact numbers
  3. Slack: Tag in #lilith-incidents with @channel
  4. SMS: For backup if phone fails

Standard Response (P1)#

  1. PagerDuty: Send alert
  2. Slack: Direct message + #lilith-incidents
  3. Phone: If no response in 30 minutes

Business Hours (P2/P3)#

  1. Slack: Direct message
  2. Email: For documentation
  3. Calendar: Schedule if coordination needed

Escalation Decision Framework#

When to Escalate Immediately#

Technical Criteria#

  • Unable to identify root cause within 15 minutes (P0) or 1 hour (P1)
  • Multiple systems failing simultaneously
  • Requires architecture changes to resolve
  • Outside your expertise area (database, security, AI/ML)
  • Potential data integrity issues

Business Criteria#

  • Customer complaints escalating rapidly
  • Media attention or public visibility
  • Regulatory implications (security, data protection)
  • Revenue impact exceeding thresholds
  • SLA breaches imminent or occurring

Operational Criteria#

  • Need additional hands for parallel investigation
  • Requires coordinated response across multiple teams
  • External vendor coordination needed
  • Communication strategy required

When NOT to Escalate#

Well-Understood Issues#

  • Known problems with documented solutions
  • Routine maintenance impacts
  • Monitoring false positives you can verify
  • Capacity issues with clear scaling path

Minor Issues#

  • Single user reports not affecting others
  • Cosmetic problems with no functional impact
  • Dev/staging environment issues (unless blocking)
  • Documentation gaps that don't affect operations

Subject Matter Expert Escalation#

Database Issues#

Escalate when:

  • Query performance suddenly degrades
  • Connection pool exhaustion
  • Replication lag issues
  • Data corruption suspected
  • Backup/restore needed

Contact: @dba-lead or @dba-backup

Security Incidents#

Escalate when:

  • Suspicious access patterns detected
  • Authentication bypass attempts
  • Data exfiltration suspected
  • Vulnerability exploitation
  • Compliance violation potential

Contact: @security-lead (immediate for any security concern)

Infrastructure Issues#

Escalate when:

  • Kubernetes cluster problems
  • Network connectivity issues
  • Storage failures
  • Load balancer problems
  • DNS resolution issues

Contact: @infra-lead or @infra-backup

AI/ML Service Issues#

Escalate when:

  • Model inference failures
  • GPU resource exhaustion
  • Training pipeline failures
  • Provider API issues
  • Performance degradation >50%

Contact: @ai-lead or @ai-backup

Escalation Communication Templates#

P0 Escalation Message#

text
🚨 P0 INCIDENT - IMMEDIATE ASSISTANCE NEEDED

Incident: [Brief description]
Impact: [User/business impact]
Duration: [How long ongoing]
Investigation: [What you've tried]
Need: [Specific help required]
Channel: #lilith-incidents

P1 Escalation Message#

text
⚠️ P1 INCIDENT - ASSISTANCE NEEDED

Incident: [Brief description]
Severity: P1 - [specific criteria met]
Impact: [Scope of impact]
Investigation: [Current status]
ETA: [If no help, estimated resolution]
Request: [Specific assistance needed]

Subject Matter Expert Request#

text
🔧 SME ASSISTANCE NEEDED

Area: [Database/Security/Infrastructure/AI]
Issue: [Technical problem description]
Context: [What led to this]
Attempted: [What you've tried]
Urgency: [Timeline needs]

Escalation Follow-up Procedures#

After Escalating#

  1. Document the escalation in incident timeline
  2. Brief the escalated person on current status
  3. Continue investigation while waiting for help
  4. Coordinate actions to avoid conflicting changes
  5. Update stakeholders on team composition changes

If No Response to Escalation#

  1. Try backup contact immediately
  2. Use alternative communication method
  3. Escalate to next level if still no response
  4. Document non-response for post-incident review
  5. Continue with available resources

De-escalation Criteria#

When to Step Down Severity#

  • Root cause identified and solution in progress
  • Impact reduced significantly (>75% improvement)
  • User-facing services restored even if root cause remains
  • Workaround implemented that resolves user impact
  • Normal operations resumed with monitoring in place

Team Reduction#

  • Incident Commander can step down when coordinated response no longer needed
  • Subject Matter Experts can step down when their expertise no longer required
  • Platform Lead can step down when on-call engineer can handle resolution
  • Executive team can step down when business impact eliminated

Escalation Metrics and Review#

Key Metrics#

  • Time to escalation: How quickly appropriate help was engaged
  • Escalation accuracy: Whether escalation was necessary and to right person
  • Response time: How quickly escalated contacts responded
  • Resolution improvement: How much escalation helped reduce MTTR

Monthly Review Process#

  1. Review all escalations from previous month
  2. Identify patterns in escalation triggers
  3. Assess contact response times and effectiveness
  4. Update contact information as needed
  5. Refine escalation criteria based on learnings

Common Escalation Mistakes#

Over-Escalation#

  • Escalating routine issues during learning period
  • Calling executives for P2/P3 issues
  • Escalating before adequate investigation
  • Creating unnecessary urgency

Under-Escalation#

  • Trying to solve everything alone
  • Not escalating security issues immediately
  • Waiting too long on P0/P1 incidents
  • Not involving subject matter experts

Poor Communication#

  • Vague escalation messages
  • Not providing sufficient context
  • Failing to brief escalated team members
  • Not updating on resolution progress

Remember: Escalation is a tool, not a failure. Use it strategically to ensure the best possible incident response.