Eve library
Lilith Platform Runbooks
Runbook
Revision 1
On this page
Status: Production Ready | Last Updated: 2024-09-15 | Version: 1.0.0
Quick Access#
| Role |
Primary |
Secondary |
Escalation |
| Platform Lead |
@platform-lead |
@platform-deputy |
@engineering-director |
| Security Lead |
@security-lead |
@security-deputy |
@ciso |
| Infrastructure Lead |
@infra-lead |
@infra-deputy |
@platform-lead |
| On-Call Engineer |
Current rotation |
Backup rotation |
@platform-lead |
Alert Severity Levels#
| Severity |
Definition |
Response Time |
Escalation |
| P0 |
Platform down, data loss, security breach |
Immediate |
15 minutes |
| P1 |
Core services degraded, user impact |
30 minutes |
1 hour |
| P2 |
Non-critical issues, limited impact |
2 hours |
4 hours |
| P3 |
Monitoring alerts, no user impact |
Next business day |
None |
Service Architecture Overview#
text
Clients β BFF β Core Services β AI Layer β Data Stores
Core Services#
- bff: Backend-for-Frontend aggregator
- auth: Identity and token management
- content: Content and course catalog
- conversation: Chat and message management
- media: Media processing and storage
- notification: Push and email dispatch
- moderation: Safety and content filtering
- ai: LLM orchestration and RAG
Infrastructure#
- Kubernetes: Multi-region deployment
- ArgoCD: GitOps deployment
- Terraform: Infrastructure as Code
- Service Mesh: mTLS, retries, circuit breakers
- Monitoring: Prometheus, Grafana, Loki, Tempo
Service Health Dashboards#
| Service |
Grafana Dashboard |
Health Endpoint |
SLO |
| BFF |
/d/bff-overview |
/v1/health |
99.9% |
| Auth |
/d/auth-overview |
/v1/health |
99.95% |
| Conversation |
/d/conversation-overview |
/v1/health |
99.9% |
| Media |
/d/media-overview |
/v1/health |
99.95% |
| AI |
/d/ai-overview |
/v1/health |
99.5% |
Communication Channels#
- #lilith-incidents - Active incident coordination
- #lilith-on-call - On-call engineer coordination
- #lilith-alerts - Automated alerting
- #lilith-deployments - Deployment notifications
Documentation Index#
On-Call Procedures#
Incident Response#
Disaster Recovery#
Operational Procedures#
Runbook Maintenance#
- Review Frequency: Monthly
- Test Frequency: Quarterly chaos engineering exercises
- Update Process: PR-based with engineering review
- Ownership: Platform Engineering team
Version History#
| Version |
Date |
Changes |
Author |
| 1.0.0 |
2024-09-15 |
Initial runbook creation |
System |
Note: These runbooks are living documents. If you encounter issues or
improvements during incident response, please create a PR with updates.