Purpose: Leadership framework for coordinating complex incidents and multi-team responses
When Incident Commander is Needed#
Automatic IC Assignment#
- P0 incidents - All severity P0 incidents require IC
- Multi-service failures - When >2 services affected
- Security incidents - Any confirmed security breach
- Extended duration - P1 incidents lasting >2 hours
IC Request Criteria#
- Resource coordination - Multiple teams needed
- External communication - Customer/media coordination required
- Complex investigation - Root cause spans multiple systems
- Business critical - Revenue or reputation impact
IC Role and Responsibilities#
Primary Responsibilities#
- Coordinate response across multiple teams and individuals
- Make decisions when technical teams disagree or are uncertain
- Manage communication both internal and external
- Track progress and ensure focus on resolution
- Escalate appropriately when additional resources needed
What IC Does NOT Do#
- Direct technical implementation (delegate to technical leads)
- Perform hands-on debugging (stay at coordination level)
- Make technical architecture decisions (facilitate SME decisions)
- Write code or execute commands (coordinate others doing this)
IC Activation Process#
Self-Activation (Experienced ICs)#
text
1. Join #lilith-incidents
2. Announce IC role assumption
3. Request current status from on-call engineer
4. Begin coordination procedures
Requested Activation#
text
1. On-call engineer requests IC in #lilith-incidents
2. Platform lead assigns IC (or assumes role)
3. IC joins incident response
4. Handoff brief from current responders
IC Announcement Template#
text
π― INCIDENT COMMANDER ACTIVE
IC: @[your-name]
Incident: [brief description]
Current Status: [summary from on-call engineer]
Next Actions: [immediate priorities]
All coordination through this channel.
Technical updates every 15 minutes.
Incident Command Structure#
Role Assignments#
text
Incident Commander (IC): Overall coordination and decisions
βββ Technical Lead: Primary investigator and implementer
βββ Communications Lead: Customer and stakeholder messaging
βββ Scribe: Timeline documentation and action tracking
βββ Subject Matter Experts: Specialized knowledge providers
Team Formation Process#
- Assess current team - Who's already responding?
- Identify gaps - What expertise is missing?
- Assign roles - Clear role definition and expectations
- Communicate structure - Ensure everyone understands reporting
Role Assignment Template#
text
INCIDENT TEAM STRUCTURE
Incident Commander: @[name]
Technical Lead: @[name] - Primary investigation
Communications: @[name] - Customer updates
Scribe: @[name] - Timeline and actions
SME Database: @[name] - Database expertise
SME Security: @[name] - Security concerns
All technical coordination through Technical Lead.
All external comms through Communications Lead.
IC Command Procedures#
Situation Assessment (First 10 minutes)#
text
1. Current Impact Assessment
- What services are affected?
- How many users impacted?
- What functionality is broken?
- Any business/revenue impact?
2. Response Status Review
- Who's currently investigating?
- What has been tried so far?
- What hypotheses exist?
- Any mitigations in place?
3. Resource Evaluation
- Do we have right expertise engaged?
- Are there capacity constraints?
- What external dependencies exist?
- Any escalation triggers approaching?
Decision-Making Framework#
When to Make Quick Decisions#
- Clear technical consensus exists among SMEs
- Low-risk mitigations that won't worsen situation
- Standard procedures apply to the situation
- Time-sensitive actions with clear benefit
When to Facilitate Discussion#
- Technical disagreement among experts
- High-risk changes that could impact other services
- Novel situations without clear precedent
- Trade-off decisions between competing priorities
Decision Documentation Template#
text
DECISION LOG - [Timestamp]
Context: [Situation requiring decision]
Options: [Alternatives considered]
Decision: [What was decided]
Rationale: [Why this option chosen]
Risk: [Potential negative consequences]
Success Criteria: [How to measure if decision worked]
Responsible: [Who will implement]
Timeline: [When decision will be evaluated]
Communication Management#
Internal Communication Rhythm#
text
Technical Updates: Every 15 minutes during active investigation
Stakeholder Updates: Every 30 minutes for P0/P1
Executive Updates: Every hour for P0, every 2 hours for P1
Team Sync: As needed for coordination
Communication Channel Management#
text
#lilith-incidents: Primary coordination
#lilith-leadership: Executive updates
#lilith-support: Customer support briefings
Bridge Line: Complex coordination (when needed)
Update Template (Every 15 minutes)#
text
INCIDENT UPDATE [#N] - [Timestamp]
Status: [Investigating/Mitigating/Resolved]
Current Focus: [Primary activity/investigation]
Progress: [What we've learned/accomplished]
Blockers: [What's preventing faster progress]
Next Steps: [Next 15-30 minutes plan]
Resources: [Team changes or needs]
ETA: [Current best estimate]
Coordination Techniques#
Parallel Investigation Management#
text
1. Divide investigation areas by expertise
2. Set regular check-in intervals (15 minutes)
3. Share findings immediately in common channel
4. Avoid duplicate work through clear assignments
5. Synthesize findings to build complete picture
Resource Conflict Resolution#
text
Scenario: Multiple urgent tasks, limited expert availability
Process:
1. List all critical tasks with estimated effort
2. Assess risk of delaying each task
3. Identify opportunities for parallelization
4. Make explicit priority decision
5. Communicate decision and rationale
6. Reassess every 30 minutes
External Vendor Coordination#
text
1. Identify which team member has vendor relationships
2. Assign single point of contact for each vendor
3. Set expectations for response time and updates
4. Escalate vendor issues through established channels
5. Document vendor response for post-incident review
Advanced IC Scenarios#
Multi-Region Outage#
text
Special Considerations:
- Time zone coordination for response teams
- Regional expertise and access requirements
- Cascading failure prevention
- Customer communication across regions
- Regulatory notification requirements
IC Actions:
1. Establish regional coordinators
2. Set up global communication rhythm
3. Coordinate cross-region mitigation
4. Manage global customer communication
5. Track region-specific recovery
Security Incident Coordination#
text
Special Considerations:
- Evidence preservation requirements
- Legal and compliance notifications
- Media and customer communication sensitivity
- Law enforcement coordination (if needed)
- Internal security protocols
IC Actions:
1. Immediately engage security lead
2. Establish evidence preservation protocols
3. Coordinate with legal team
4. Manage information sharing restrictions
5. Balance transparency with security needs
Vendor-Caused Outage#
text
Special Considerations:
- Limited direct control over resolution
- Customer communication challenges
- SLA and liability concerns
- Alternative provider activation
- Coordination with vendor incident response
IC Actions:
1. Establish vendor communication channel
2. Activate alternative providers if available
3. Coordinate workaround development
4. Manage customer expectations about ETA
5. Document vendor performance for review
Cascading Failure Management#
text
Special Considerations:
- Multiple simultaneous incidents
- Complex interdependencies
- Resource prioritization challenges
- Communication complexity
- Risk of additional failures
IC Actions:
1. Map service dependencies quickly
2. Prioritize based on user impact
3. Implement circuit breakers to prevent spread
4. Coordinate staged recovery
5. Monitor for additional cascading effects
IC Handoff Procedures#
When to Hand Off IC Role#
- Extended duration - IC fatigue after 4-6 hours
- Shift change - Regular rotation schedules
- Expertise mismatch - Need IC with specific domain knowledge
- Escalation - Senior leadership assumes IC role
- Resolution transition - Moving from response to recovery
IC Handoff Process#
text
1. Brief Status Summary (5 minutes)
- Current incident state
- Team composition and roles
- Recent decisions and rationale
- Key next steps and timeline
2. Context Transfer (10 minutes)
- Root cause investigation status
- Previous mitigation attempts
- External dependencies and contacts
- Communication commitments made
3. Team Introduction (5 minutes)
- Introduce new IC to all team members
- Confirm role transitions if any
- Update communication channels
- Verify access and tool availability
4. Decision Authority Transfer
- Explicit announcement of IC change
- Confirm with technical leads
- Update stakeholder communications
- Transfer incident management tools
Handoff Documentation Template#
text
IC HANDOFF REPORT
Previous IC: @[name] ([start-time] to [end-time])
New IC: @[name] (starting [time])
Current Status:
- Incident State: [investigating/mitigating/monitoring]
- User Impact: [current impact level]
- Team Size: [number of people engaged]
- Investigation Focus: [primary areas]
Key Context:
- Root Cause: [current understanding]
- Attempted Fixes: [what's been tried]
- Current Theory: [working hypothesis]
- Blockers: [what's preventing progress]
Team Composition:
- Technical Lead: @[name]
- Communications: @[name]
- Scribe: @[name]
- SMEs: @[names and specialties]
Commitments Made:
- Customer Updates: [frequency and next due]
- Executive Updates: [frequency and next due]
- Vendor Coordination: [active vendor calls/tickets]
Next Critical Actions:
- [Priority 1 action with timeline]
- [Priority 2 action with timeline]
- [Priority 3 action with timeline]
Contact Info:
- Previous IC: [phone/slack for questions]
- Available Until: [when previous IC goes offline]
IC Performance and Improvement#
IC Success Metrics#
- Time to team formation - How quickly was appropriate expertise engaged?
- Decision velocity - Were decisions made quickly when needed?
- Communication effectiveness - Did stakeholders have appropriate information?
- Resource utilization - Were people used effectively without burnout?
- Escalation appropriateness - Was escalation timed correctly?
Common IC Pitfalls#
Over-Managing#
- Getting too involved in technical details
- Making decisions that should be delegated
- Not trusting SME expertise
- Micromanaging investigation approaches
Under-Managing#
- Not asking for status updates
- Failing to make necessary decisions
- Poor communication with stakeholders
- Not escalating when appropriate
Communication Issues#
- Too frequent updates without meaningful progress
- Too infrequent updates leaving stakeholders uninformed
- Technical jargon in business communications
- Inconsistent messaging across channels
IC Development and Training#
Mandatory Training#
- Incident command fundamentals
- Communication and leadership during crisis
- Platform architecture overview
- Escalation procedures and contacts
Recommended Experience#
- On-call rotation experience
- Previous technical lead role in incidents
- Cross-team project coordination
- Customer communication experience
Ongoing Development#
- Monthly IC review sessions
- Quarterly incident simulations
- Cross-training with other ICs
- Post-incident coaching and feedback
Remember: As IC, your job is to coordinate, not to fix. Trust your technical teams, make decisions when needed, and keep everyone informed.