Eve library

Incident Commander Guide

Runbook

Revision 1

On this page

Purpose: Leadership framework for coordinating complex incidents and multi-team responses

When Incident Commander is Needed#

Automatic IC Assignment#

  • P0 incidents - All severity P0 incidents require IC
  • Multi-service failures - When >2 services affected
  • Security incidents - Any confirmed security breach
  • Extended duration - P1 incidents lasting >2 hours

IC Request Criteria#

  • Resource coordination - Multiple teams needed
  • External communication - Customer/media coordination required
  • Complex investigation - Root cause spans multiple systems
  • Business critical - Revenue or reputation impact

IC Role and Responsibilities#

Primary Responsibilities#

  1. Coordinate response across multiple teams and individuals
  2. Make decisions when technical teams disagree or are uncertain
  3. Manage communication both internal and external
  4. Track progress and ensure focus on resolution
  5. Escalate appropriately when additional resources needed

What IC Does NOT Do#

  • Direct technical implementation (delegate to technical leads)
  • Perform hands-on debugging (stay at coordination level)
  • Make technical architecture decisions (facilitate SME decisions)
  • Write code or execute commands (coordinate others doing this)

IC Activation Process#

Self-Activation (Experienced ICs)#

text
1. Join #lilith-incidents
2. Announce IC role assumption
3. Request current status from on-call engineer
4. Begin coordination procedures

Requested Activation#

text
1. On-call engineer requests IC in #lilith-incidents
2. Platform lead assigns IC (or assumes role)
3. IC joins incident response
4. Handoff brief from current responders

IC Announcement Template#

text
🎯 INCIDENT COMMANDER ACTIVE

IC: @[your-name]
Incident: [brief description]
Current Status: [summary from on-call engineer]
Next Actions: [immediate priorities]

All coordination through this channel.
Technical updates every 15 minutes.

Incident Command Structure#

Role Assignments#

text
Incident Commander (IC): Overall coordination and decisions
β”œβ”€β”€ Technical Lead: Primary investigator and implementer
β”œβ”€β”€ Communications Lead: Customer and stakeholder messaging
β”œβ”€β”€ Scribe: Timeline documentation and action tracking
└── Subject Matter Experts: Specialized knowledge providers

Team Formation Process#

  1. Assess current team - Who's already responding?
  2. Identify gaps - What expertise is missing?
  3. Assign roles - Clear role definition and expectations
  4. Communicate structure - Ensure everyone understands reporting

Role Assignment Template#

text
INCIDENT TEAM STRUCTURE

Incident Commander: @[name]
Technical Lead: @[name] - Primary investigation
Communications: @[name] - Customer updates
Scribe: @[name] - Timeline and actions
SME Database: @[name] - Database expertise
SME Security: @[name] - Security concerns

All technical coordination through Technical Lead.
All external comms through Communications Lead.

IC Command Procedures#

Situation Assessment (First 10 minutes)#

text
1. Current Impact Assessment
   - What services are affected?
   - How many users impacted?
   - What functionality is broken?
   - Any business/revenue impact?

2. Response Status Review
   - Who's currently investigating?
   - What has been tried so far?
   - What hypotheses exist?
   - Any mitigations in place?

3. Resource Evaluation
   - Do we have right expertise engaged?
   - Are there capacity constraints?
   - What external dependencies exist?
   - Any escalation triggers approaching?

Decision-Making Framework#

When to Make Quick Decisions#

  • Clear technical consensus exists among SMEs
  • Low-risk mitigations that won't worsen situation
  • Standard procedures apply to the situation
  • Time-sensitive actions with clear benefit

When to Facilitate Discussion#

  • Technical disagreement among experts
  • High-risk changes that could impact other services
  • Novel situations without clear precedent
  • Trade-off decisions between competing priorities

Decision Documentation Template#

text
DECISION LOG - [Timestamp]

Context: [Situation requiring decision]
Options: [Alternatives considered]
Decision: [What was decided]
Rationale: [Why this option chosen]
Risk: [Potential negative consequences]
Success Criteria: [How to measure if decision worked]
Responsible: [Who will implement]
Timeline: [When decision will be evaluated]

Communication Management#

Internal Communication Rhythm#

text
Technical Updates: Every 15 minutes during active investigation
Stakeholder Updates: Every 30 minutes for P0/P1
Executive Updates: Every hour for P0, every 2 hours for P1
Team Sync: As needed for coordination

Communication Channel Management#

text
#lilith-incidents: Primary coordination
#lilith-leadership: Executive updates
#lilith-support: Customer support briefings
Bridge Line: Complex coordination (when needed)

Update Template (Every 15 minutes)#

text
INCIDENT UPDATE [#N] - [Timestamp]

Status: [Investigating/Mitigating/Resolved]
Current Focus: [Primary activity/investigation]
Progress: [What we've learned/accomplished]
Blockers: [What's preventing faster progress]
Next Steps: [Next 15-30 minutes plan]
Resources: [Team changes or needs]
ETA: [Current best estimate]

Coordination Techniques#

Parallel Investigation Management#

text
1. Divide investigation areas by expertise
2. Set regular check-in intervals (15 minutes)
3. Share findings immediately in common channel
4. Avoid duplicate work through clear assignments
5. Synthesize findings to build complete picture

Resource Conflict Resolution#

text
Scenario: Multiple urgent tasks, limited expert availability
Process:
1. List all critical tasks with estimated effort
2. Assess risk of delaying each task
3. Identify opportunities for parallelization
4. Make explicit priority decision
5. Communicate decision and rationale
6. Reassess every 30 minutes

External Vendor Coordination#

text
1. Identify which team member has vendor relationships
2. Assign single point of contact for each vendor
3. Set expectations for response time and updates
4. Escalate vendor issues through established channels
5. Document vendor response for post-incident review

Advanced IC Scenarios#

Multi-Region Outage#

text
Special Considerations:
- Time zone coordination for response teams
- Regional expertise and access requirements
- Cascading failure prevention
- Customer communication across regions
- Regulatory notification requirements

IC Actions:
1. Establish regional coordinators
2. Set up global communication rhythm
3. Coordinate cross-region mitigation
4. Manage global customer communication
5. Track region-specific recovery

Security Incident Coordination#

text
Special Considerations:
- Evidence preservation requirements
- Legal and compliance notifications
- Media and customer communication sensitivity
- Law enforcement coordination (if needed)
- Internal security protocols

IC Actions:
1. Immediately engage security lead
2. Establish evidence preservation protocols
3. Coordinate with legal team
4. Manage information sharing restrictions
5. Balance transparency with security needs

Vendor-Caused Outage#

text
Special Considerations:
- Limited direct control over resolution
- Customer communication challenges
- SLA and liability concerns
- Alternative provider activation
- Coordination with vendor incident response

IC Actions:
1. Establish vendor communication channel
2. Activate alternative providers if available
3. Coordinate workaround development
4. Manage customer expectations about ETA
5. Document vendor performance for review

Cascading Failure Management#

text
Special Considerations:
- Multiple simultaneous incidents
- Complex interdependencies
- Resource prioritization challenges
- Communication complexity
- Risk of additional failures

IC Actions:
1. Map service dependencies quickly
2. Prioritize based on user impact
3. Implement circuit breakers to prevent spread
4. Coordinate staged recovery
5. Monitor for additional cascading effects

IC Handoff Procedures#

When to Hand Off IC Role#

  • Extended duration - IC fatigue after 4-6 hours
  • Shift change - Regular rotation schedules
  • Expertise mismatch - Need IC with specific domain knowledge
  • Escalation - Senior leadership assumes IC role
  • Resolution transition - Moving from response to recovery

IC Handoff Process#

text
1. Brief Status Summary (5 minutes)
   - Current incident state
   - Team composition and roles
   - Recent decisions and rationale
   - Key next steps and timeline

2. Context Transfer (10 minutes)
   - Root cause investigation status
   - Previous mitigation attempts
   - External dependencies and contacts
   - Communication commitments made

3. Team Introduction (5 minutes)
   - Introduce new IC to all team members
   - Confirm role transitions if any
   - Update communication channels
   - Verify access and tool availability

4. Decision Authority Transfer
   - Explicit announcement of IC change
   - Confirm with technical leads
   - Update stakeholder communications
   - Transfer incident management tools

Handoff Documentation Template#

text
IC HANDOFF REPORT

Previous IC: @[name] ([start-time] to [end-time])
New IC: @[name] (starting [time])

Current Status:
- Incident State: [investigating/mitigating/monitoring]
- User Impact: [current impact level]
- Team Size: [number of people engaged]
- Investigation Focus: [primary areas]

Key Context:
- Root Cause: [current understanding]
- Attempted Fixes: [what's been tried]
- Current Theory: [working hypothesis]
- Blockers: [what's preventing progress]

Team Composition:
- Technical Lead: @[name]
- Communications: @[name]
- Scribe: @[name]
- SMEs: @[names and specialties]

Commitments Made:
- Customer Updates: [frequency and next due]
- Executive Updates: [frequency and next due]
- Vendor Coordination: [active vendor calls/tickets]

Next Critical Actions:
- [Priority 1 action with timeline]
- [Priority 2 action with timeline]
- [Priority 3 action with timeline]

Contact Info:
- Previous IC: [phone/slack for questions]
- Available Until: [when previous IC goes offline]

IC Performance and Improvement#

IC Success Metrics#

  • Time to team formation - How quickly was appropriate expertise engaged?
  • Decision velocity - Were decisions made quickly when needed?
  • Communication effectiveness - Did stakeholders have appropriate information?
  • Resource utilization - Were people used effectively without burnout?
  • Escalation appropriateness - Was escalation timed correctly?

Common IC Pitfalls#

Over-Managing#

  • Getting too involved in technical details
  • Making decisions that should be delegated
  • Not trusting SME expertise
  • Micromanaging investigation approaches

Under-Managing#

  • Not asking for status updates
  • Failing to make necessary decisions
  • Poor communication with stakeholders
  • Not escalating when appropriate

Communication Issues#

  • Too frequent updates without meaningful progress
  • Too infrequent updates leaving stakeholders uninformed
  • Technical jargon in business communications
  • Inconsistent messaging across channels

IC Development and Training#

Mandatory Training#

  • Incident command fundamentals
  • Communication and leadership during crisis
  • Platform architecture overview
  • Escalation procedures and contacts
  • On-call rotation experience
  • Previous technical lead role in incidents
  • Cross-team project coordination
  • Customer communication experience

Ongoing Development#

  • Monthly IC review sessions
  • Quarterly incident simulations
  • Cross-training with other ICs
  • Post-incident coaching and feedback

Remember: As IC, your job is to coordinate, not to fix. Trust your technical teams, make decisions when needed, and keep everyone informed.