Eve library

Incident Response Plan

Runbook

Revision 1

On this page

Lilith Platform Smart Contract Security#

Version 1.0 | Last Updated: 2024#


Table of Contents#

  1. Incident Severity Classification
  2. Response Team & Roles
  3. Detection & Alerting
  4. Response Procedures
  5. Communication Protocols
  6. Post-Incident Procedures
  7. User Compensation Process
  8. Tabletop Exercises
  9. On-Call Rotation

Incident Severity Classification#

P0 - CRITICAL (Active Exploit)#

Definition: Active exploitation in progress, funds at immediate risk, or complete system compromise.

Examples:

  • Ongoing fund drain from contracts
  • Private key compromise
  • Smart contract logic exploit being actively used
  • Flash loan attack successfully bypassing guards
  • Unauthorized ownership transfer

Response Time: IMMEDIATE (< 5 minutes)

Actions:

  1. Emergency pause ALL affected contracts
  2. Alert entire security team via emergency channel
  3. Initiate war room immediately
  4. Notify legal team
  5. Prepare public communication

Escalation: CEO, CTO, Legal, PR within 15 minutes


P1 - HIGH (Vulnerability Discovered)#

Definition: Critical vulnerability identified but not yet exploited. Immediate mitigation required.

Examples:

  • Critical vulnerability found in audit or bug bounty
  • Potential exploit vector identified in production
  • Suspicious transaction patterns detected
  • Failed flash loan attack (guard triggered)
  • Unauthorized role changes detected

Response Time: < 1 hour

Actions:

  1. Assess exploitability and impact
  2. Consider preventive pause if high risk
  3. Begin mitigation development
  4. Monitor for exploitation attempts
  5. Prepare patch deployment

Escalation: Security Lead, Smart Contract Team Lead within 30 minutes


P2 - MEDIUM (Issue Identified)#

Definition: Non-critical issue or vulnerability with no immediate risk. Scheduled mitigation.

Examples:

  • Medium-severity vulnerability in non-critical path
  • Gas optimization issues causing failed transactions
  • UI/frontend issues affecting user experience
  • Suboptimal contract configuration

Response Time: < 24 hours (next business day)

Actions:

  1. Document issue thoroughly
  2. Assess priority for next release
  3. Develop and test fix
  4. Schedule deployment

Escalation: Team Lead within 4 hours


P3 - LOW (Minor Issue)#

Definition: Minor issues, optimizations, or improvements. No security impact.

Examples:

  • Documentation updates needed
  • Minor UI improvements
  • Non-critical gas optimizations
  • Feature enhancement requests

Response Time: Next sprint/release cycle

Actions:

  1. Create tracking ticket
  2. Prioritize in backlog
  3. Address in regular development cycle

Response Team & Roles#

Incident Commander (IC)#

  • Responsibility: Overall coordination of incident response
  • Authority: Can authorize emergency actions including pause
  • Primary: Security Lead
  • Backup: CTO

Technical Lead#

  • Responsibility: Technical analysis and mitigation
  • Authority: Direct smart contract and infrastructure changes
  • Primary: Smart Contract Team Lead
  • Backup: Senior Smart Contract Engineer

Communications Lead#

  • Responsibility: Internal and external communications
  • Authority: Approve and publish public statements
  • Primary: PR/Communications Manager
  • Backup: CEO

Operations Lead#

  • Responsibility: Monitor systems and execute commands
  • Authority: Execute authorized commands on infrastructure
  • Primary: DevOps Lead
  • Backup: Senior DevOps Engineer

Detection & Alerting#

Monitoring Systems#

System Coverage Alert Destination
OpenZeppelin Defender On-chain events PagerDuty, Slack
Forta Suspicious patterns PagerDuty, Slack
The Graph Historical analysis Grafana, Email
Internal Monitor Real-time events Slack #alerts

Alert Priority Mapping#

Severity PagerDuty Slack Email SMS
P0 Immediate #incidents Yes Yes
P1 High #alerts Yes On-call
P2 Low #monitoring Yes No
P3 None #monitoring No No

Response Procedures#

P0 - Active Exploit Response#

text
TIME    ACTION
0:00    Alert received
0:02    Incident Commander confirms P0 severity
0:05    Emergency pause executed on all affected contracts
0:10    War room opened (video call link sent)
0:15    Initial assessment complete
0:30    Stakeholders briefed
1:00    Public holding statement published
2:00    Detailed analysis complete
4:00    Remediation plan drafted
24:00   Post-mortem meeting scheduled

Emergency Pause Authorization#

The following roles can authorize emergency pause:

  1. CEO - Full authority
  2. CTO - Full authority
  3. Security Lead - Full authority for P0/P1
  4. Smart Contract Lead - Authority for P0 only
  5. On-Call Engineer - Authority for P0 only (with immediate escalation)

Emergency Pause Procedure#

bash
# 1. Connect to production network
export NETWORK=mainnet

# 2. Execute emergency pause via Safe
# Option A: Using Defender Relay
npx hardhat run scripts/emergency-pause.ts --network $NETWORK

# Option B: Direct Safe transaction
# Submit pause transaction to Safe multisig
# Collect required signatures
# Execute transaction

# 3. Verify pause
npx hardhat run scripts/verify-pause.ts --network $NETWORK

# 4. Notify all stakeholders
# Send to: #incidents, leadership, legal

P1 - Vulnerability Response#

text
TIME    ACTION
0:00    Alert/report received
0:15    Initial triage by on-call
0:30    Severity confirmed by Security Lead
1:00    Assessment meeting with relevant engineers
2:00    Mitigation strategy decided
4:00    Mitigation developed (if applicable)
8:00    Mitigation tested on testnet
24:00   Fix deployed or preventive measures in place
48:00   Full resolution deployed
72:00   Post-mortem complete

Communication Protocols#

Internal Channels#

Channel Purpose Audience
#incidents Active P0/P1 coordination Response team
#alerts All security alerts Engineering
#monitoring Routine monitoring DevOps
War Room (Video) Real-time coordination Response team

External Channels#

Channel Purpose Timing
Twitter/X User updates Within 1 hour of P0
Discord Community updates Within 2 hours
Blog Detailed updates Within 24 hours
Email Direct user notification As needed

Communication Templates#

Holding Statement (P0)#

text
[SECURITY NOTICE]

We are currently investigating a security incident affecting [CONTRACT/SERVICE].

For your protection, we have temporarily paused [affected services].

Your funds are [safe/being assessed]. We will provide updates every [30 minutes/hour].

Status page: [link]
Discord: [link]

We appreciate your patience.

Resolution Statement#

text
[SECURITY UPDATE]

The security incident reported on [DATE] has been resolved.

Summary:
- Issue: [Brief description]
- Impact: [Number of affected users, funds at risk]
- Resolution: [Actions taken]
- Timeline: [Key events]

Affected users will be [compensated/contacted] within [timeframe].

Detailed post-mortem: [link]

Post-Incident Procedures#

Post-Mortem Template#

markdown
# Incident Post-Mortem: [INCIDENT-ID]

## Summary
- **Incident Date:** YYYY-MM-DD
- **Severity:** P0/P1/P2
- **Duration:** X hours
- **Impact:** [Description]

## Timeline
| Time (UTC) | Event |
|------------|-------|
| HH:MM | Alert received |
| HH:MM | Incident confirmed |
| HH:MM | Mitigation started |
| HH:MM | Resolution deployed |
| HH:MM | All clear declared |

## Root Cause
[Detailed technical explanation]

## Impact Assessment
- Users affected: X
- Funds at risk: X ETH
- Actual losses: X ETH
- Transactions affected: X

## Response Evaluation
### What Went Well
- [Item 1]
- [Item 2]

### What Could Improve
- [Item 1]
- [Item 2]

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| [Action] | [Name] | [Date] | [Status] |

## Lessons Learned
[Key takeaways for future prevention]

Post-Mortem Meeting Agenda#

  1. Timeline review (15 min)
  2. Root cause analysis (20 min)
  3. Response effectiveness (15 min)
  4. Prevention measures (20 min)
  5. Action item assignment (10 min)

User Compensation Process#

Eligibility Assessment#

  1. Verify affected users from on-chain data

    • Query affected transactions
    • Calculate individual losses
    • Verify wallet ownership
  2. Compensation Categories

Category Compensation Timeline
Direct fund loss 100% of verified loss 7 days
Failed transaction fees 100% of gas spent 7 days
Opportunity cost Case by case 14 days
Goodwill Discretionary 30 days
  1. Compensation Mechanisms
  • On-chain refund: Direct transfer to affected wallets
  • Claim portal: User-initiated claims with verification
  • Airdrop: For smaller amounts to many users

Claim Verification#

solidity
// Claims require:
// 1. Proof of affected transaction
// 2. Wallet signature verification
// 3. Amount calculation verification
// 4. Duplicate claim prevention

Tabletop Exercises#

Quarterly Exercise Schedule#

Quarter Scenario Focus Area
Q1 Flash loan attack Detection & pause
Q2 Private key compromise Multi-sig recovery
Q3 Smart contract exploit Patching & deployment
Q4 Full incident simulation End-to-end response

Exercise Format#

  1. Preparation (1 week before)

    • Define scenario
    • Prepare injects
    • Brief exercise controller
  2. Execution (2-3 hours)

    • Scenario introduction (15 min)
    • Response simulation (90 min)
    • Debrief (45 min)
  3. Follow-up (1 week after)

    • Document lessons learned
    • Update procedures
    • Track action items

Sample Scenarios#

Scenario 1: Flash Loan Attack#

text
INJECT 1: Forta bot detects unusual transaction pattern
INJECT 2: User reports unexpected fund movement
INJECT 3: Attacker drains 50 ETH from AccessToken
INJECT 4: Media inquiry received

Scenario 2: Key Compromise#

text
INJECT 1: Unusual Safe transaction submitted
INJECT 2: Team member reports phishing attempt
INJECT 3: Unauthorized ownership transfer detected
INJECT 4: Attacker posts on social media

On-Call Rotation#

Schedule#

  • Primary: 1 week rotation
  • Secondary: 1 week rotation (shadow)
  • Coverage: 24/7/365
  • Handoff: Monday 10:00 UTC

On-Call Responsibilities#

  1. Monitor #alerts channel
  2. Respond to PagerDuty within 15 minutes
  3. Escalate P0/P1 immediately
  4. Document all incidents
  5. Brief next on-call at handoff

Escalation Path#

text
On-Call Engineer (15 min)
    ↓
Security Lead (30 min)
    ↓
CTO (1 hour)
    ↓
CEO (as needed)

On-Call Tools#

  • PagerDuty mobile app configured
  • Safe wallet access (view only)
  • Defender dashboard access
  • Slack mobile with #incidents notifications
  • War room video call link
  • Emergency contact list

Appendices#

A. Emergency Contact List#

Role Primary Secondary
CEO [REDACTED] [REDACTED]
CTO [REDACTED] [REDACTED]
Security Lead [REDACTED] [REDACTED]
Legal Counsel [REDACTED] [REDACTED]
PR Lead [REDACTED] [REDACTED]

B. External Contacts#

Organization Contact Purpose
Trail of Bits [REDACTED] Emergency audit
Chainalysis [REDACTED] Fund tracing
OpenZeppelin [REDACTED] Technical support
Legal Firm [REDACTED] Legal counsel

C. Quick Reference Card#

text
P0 RESPONSE CHECKLIST
□ Confirm severity with IC
□ Execute emergency pause
□ Open war room
□ Notify stakeholders
□ Prepare holding statement
□ Begin root cause analysis
□ Document timeline
□ Prepare user communication

This document is reviewed and updated quarterly. Last review: [DATE]