Eve library

Lilith Platform Runbooks

Runbook

Revision 1

On this page

Status: Production Ready | Last Updated: 2024-09-15 | Version: 1.0.0

Quick Access#

Emergency Contacts#

Role Primary Secondary Escalation
Platform Lead @platform-lead @platform-deputy @engineering-director
Security Lead @security-lead @security-deputy @ciso
Infrastructure Lead @infra-lead @infra-deputy @platform-lead
On-Call Engineer Current rotation Backup rotation @platform-lead

Alert Severity Levels#

Severity Definition Response Time Escalation
P0 Platform down, data loss, security breach Immediate 15 minutes
P1 Core services degraded, user impact 30 minutes 1 hour
P2 Non-critical issues, limited impact 2 hours 4 hours
P3 Monitoring alerts, no user impact Next business day None

Service Architecture Overview#

text
Clients β†’ BFF β†’ Core Services β†’ AI Layer β†’ Data Stores

Core Services#

  • bff: Backend-for-Frontend aggregator
  • auth: Identity and token management
  • content: Content and course catalog
  • conversation: Chat and message management
  • media: Media processing and storage
  • notification: Push and email dispatch
  • moderation: Safety and content filtering
  • ai: LLM orchestration and RAG

Infrastructure#

  • Kubernetes: Multi-region deployment
  • ArgoCD: GitOps deployment
  • Terraform: Infrastructure as Code
  • Service Mesh: mTLS, retries, circuit breakers
  • Monitoring: Prometheus, Grafana, Loki, Tempo

Service Health Dashboards#

Service Grafana Dashboard Health Endpoint SLO
BFF /d/bff-overview /v1/health 99.9%
Auth /d/auth-overview /v1/health 99.95%
Conversation /d/conversation-overview /v1/health 99.9%
Media /d/media-overview /v1/health 99.95%
AI /d/ai-overview /v1/health 99.5%

Communication Channels#

  • #lilith-incidents - Active incident coordination
  • #lilith-on-call - On-call engineer coordination
  • #lilith-alerts - Automated alerting
  • #lilith-deployments - Deployment notifications

Documentation Index#

On-Call Procedures#

Incident Response#

Disaster Recovery#

Operational Procedures#

Runbook Maintenance#

  • Review Frequency: Monthly
  • Test Frequency: Quarterly chaos engineering exercises
  • Update Process: PR-based with engineering review
  • Ownership: Platform Engineering team

Version History#

Version Date Changes Author
1.0.0 2024-09-15 Initial runbook creation System

Note: These runbooks are living documents. If you encounter issues or improvements during incident response, please create a PR with updates.