# Lilith Platform Runbooks

**Status**: Production Ready | **Last Updated**: 2024-09-15 | **Version**: 1.0.0

## Quick Access

- 🚨 **[Incident Response](incidents/incident-response.md)** - Start here during
  incidents
- 📞 **[On-Call Guide](on-call/on-call-guide.md)** - On-call engineer procedures
- 🔄 **[Disaster Recovery](disaster-recovery/dr-playbooks.md)** - Business
  continuity procedures

## Emergency Contacts

| Role                | Primary          | Secondary        | Escalation            |
| ------------------- | ---------------- | ---------------- | --------------------- |
| Platform Lead       | @platform-lead   | @platform-deputy | @engineering-director |
| Security Lead       | @security-lead   | @security-deputy | @ciso                 |
| Infrastructure Lead | @infra-lead      | @infra-deputy    | @platform-lead        |
| On-Call Engineer    | Current rotation | Backup rotation  | @platform-lead        |

## Alert Severity Levels

| Severity | Definition                                | Response Time     | Escalation |
| -------- | ----------------------------------------- | ----------------- | ---------- |
| **P0**   | Platform down, data loss, security breach | Immediate         | 15 minutes |
| **P1**   | Core services degraded, user impact       | 30 minutes        | 1 hour     |
| **P2**   | Non-critical issues, limited impact       | 2 hours           | 4 hours    |
| **P3**   | Monitoring alerts, no user impact         | Next business day | None       |

## Service Architecture Overview

```
Clients → BFF → Core Services → AI Layer → Data Stores
```

### Core Services

- **bff**: Backend-for-Frontend aggregator
- **auth**: Identity and token management
- **content**: Content and course catalog
- **conversation**: Chat and message management
- **media**: Media processing and storage
- **notification**: Push and email dispatch
- **moderation**: Safety and content filtering
- **ai**: LLM orchestration and RAG

### Infrastructure

- **Kubernetes**: Multi-region deployment
- **ArgoCD**: GitOps deployment
- **Terraform**: Infrastructure as Code
- **Service Mesh**: mTLS, retries, circuit breakers
- **Monitoring**: Prometheus, Grafana, Loki, Tempo

## Service Health Dashboards

| Service      | Grafana Dashboard          | Health Endpoint | SLO    |
| ------------ | -------------------------- | --------------- | ------ |
| BFF          | `/d/bff-overview`          | `/v1/health`    | 99.9%  |
| Auth         | `/d/auth-overview`         | `/v1/health`    | 99.95% |
| Conversation | `/d/conversation-overview` | `/v1/health`    | 99.9%  |
| Media        | `/d/media-overview`        | `/v1/health`    | 99.95% |
| AI           | `/d/ai-overview`           | `/v1/health`    | 99.5%  |

## Communication Channels

- **#lilith-incidents** - Active incident coordination
- **#lilith-on-call** - On-call engineer coordination
- **#lilith-alerts** - Automated alerting
- **#lilith-deployments** - Deployment notifications

## Documentation Index

### On-Call Procedures

- [On-Call Guide](on-call/on-call-guide.md) - Complete on-call procedures
- [Handoff Checklist](on-call/handoff-checklist.md) - Shift transition
  procedures
- [Escalation Matrix](on-call/escalation-matrix.md) - When and how to escalate

### Incident Response

- [Incident Response Process](incidents/incident-response.md) - Step-by-step
  incident handling
- [Incident Commander Guide](incidents/incident-commander.md) - Leadership
  during incidents
- [Post-Incident Review](incidents/post-incident-review.md) - Learning from
  incidents
- [Communication Templates](incidents/communication-templates.md) -
  Customer/stakeholder messaging

### Disaster Recovery

- [DR Playbooks Overview](disaster-recovery/dr-playbooks.md) - Business
  continuity procedures
- [Database Recovery](disaster-recovery/database-recovery.md) - Data restoration
  procedures
- [Service Recovery](disaster-recovery/service-recovery.md) - Application
  recovery procedures
- [Network Recovery](disaster-recovery/network-recovery.md) - Infrastructure
  recovery procedures

### Operational Procedures

- [Deployment Procedures](procedures/deployment.md) - Safe deployment practices
- [Monitoring Setup](procedures/monitoring.md) - Observability configuration
- [Security Procedures](procedures/security.md) - Security incident handling
- [Capacity Planning](procedures/capacity-planning.md) - Resource management
- [Type Declaration Artifacts](procedures/types-artifacts.md) - Policy for
  generated `.d.ts` files

## Runbook Maintenance

- **Review Frequency**: Monthly
- **Test Frequency**: Quarterly chaos engineering exercises
- **Update Process**: PR-based with engineering review
- **Ownership**: Platform Engineering team

## Version History

| Version | Date       | Changes                  | Author |
| ------- | ---------- | ------------------------ | ------ |
| 1.0.0   | 2024-09-15 | Initial runbook creation | System |

---

**Note**: These runbooks are living documents. If you encounter issues or
improvements during incident response, please create a PR with updates.
