# Security Incident Response Runbooks

## Threat Detection & Response Procedures

### High-Severity Anomaly Response

#### Detection Criteria
- Security event spike > 3-sigma deviation from baseline
- Critical severity events > threshold
- Multiple simultaneous threat types

#### Response Procedure
1. **Immediate Assessment** (0-5 minutes)
   - Access security dashboard: `GET /v1/security/dashboard`
   - Identify affected services and threat types
   - Check system health: `GET /v1/health`

2. **Threat Analysis** (5-15 minutes)
   - Correlate with application logs
   - Check rate limiter telemetry
   - Analyze traffic patterns in WAF/CDN

3. **Containment** (15-30 minutes)
   - Apply rate limiting if needed
   - Block malicious IPs via WAF
   - Scale affected services if under load

4. **Investigation** (30-60 minutes)
   - Deep dive into affected user accounts
   - Check for credential stuffing patterns
   - Analyze attack vectors and entry points

#### Escalation Criteria
- Critical severity events > 10/hour
- Multiple service compromise indicators
- Data exfiltration attempts detected

### Authentication Failure Investigation

#### Common Patterns
- **Credential Stuffing**: High volume auth failures from multiple IPs
- **Brute Force**: Repeated failures from single IP
- **Account Takeover**: Successful auth after multiple failures

#### Investigation Steps
1. Query auth service logs: `kubectl logs -n lilith deployment/auth-service`
2. Check rate limiting metrics
3. Analyze IP geolocation patterns
4. Review affected user accounts

#### Mitigation Actions
- Temporarily block suspicious IPs
- Force password reset for affected accounts
- Enable additional MFA requirements
- Update rate limiting thresholds

### Safety Policy Violations

#### Detection
- `safety.policy_violation` events
- `moderation.flagged` content
- Persona boundary violations

#### Response Process
1. Review flagged content in moderation queue
2. Analyze user interaction patterns
3. Update safety filters if needed
4. Escalate to safety team for review

## Automated Response Triggers

### High-Priority Alerts
- Critical security events: Immediate PagerDuty alert
- Anomaly detection: Slack notification
- Service health degradation: Auto-scaling trigger

### Response Automation
```javascript
// Example automated response configuration
const responseConfig = {
  'rate_limit_exceeded': {
    severity: 'medium',
    action: 'increase_rate_limits',
    cooldown: '5m'
  },
  'jailbreak_attempt': {
    severity: 'high',
    action: 'alert_security_team',
    escalation: 'immediate'
  },
  'auth_failed': {
    severity: 'low',
    action: 'track_ip_patterns',
    threshold: 50
  }
};
```

## Monitoring Commands

### Dashboard Access
```bash
# Security dashboard
curl ${OBSERVABILITY_URL}/v1/security/dashboard

# Raw security events
curl ${OBSERVABILITY_URL}/v1/security/events?timeframe=1h

# Anomaly details
curl ${OBSERVABILITY_URL}/v1/security/anomalies
```

### Kubernetes Investigation
```bash
# Check security-related pods
kubectl get pods -n lilith -l security=monitoring

# View security event logs
kubectl logs -n monitoring deployment/observability-service | grep security

# Check network policies
kubectl get networkpolicies -n lilith
```

---
*Generated automatically by security-docs-generator.js*
*Last updated: 2025-09-27T18:05:41.712Z*
