Eve library

Redis Operations & Recovery Runbook

Runbook

Revision 1

On this page

Purpose: Operational procedures for Redis infrastructure including configuration, monitoring, troubleshooting, and disaster recovery.

Redis Architecture Overview#

Development Environment#

text
Redis Configuration:
├── Image: redis:7-alpine
├── Port: 6380 (external) → 6379 (internal)
├── Memory: 512MB max (1GB container limit)
├── Persistence: AOF + RDB snapshots
└── Monitoring: redis-exporter → Prometheus

Persistence Configuration:
├── AOF (Append-Only File): Enabled with everysec fsync
└── RDB Snapshots:
    ├── Save after 900s if ≥1 key changed
    ├── Save after 300s if ≥10 keys changed
    └── Save after 60s if ≥10000 keys changed

Services Using Redis#

Service Purpose Data Type
auth Session management Sessions, tokens
conversation Real-time messaging Message cache
notification Push notification queue Notification queue
moderation Rate limiting Counters
ai Inference caching Embeddings, responses
tts/stt Audio caching Audio segments
ingestion Job queue Processing queue

Configuration Details#

Docker Compose Configuration#

yaml
redis:
  image: redis:7-alpine
  ports:
    - '6380:6379'
  volumes:
    - redis-data:/data
  command: >
    redis-server --appendonly yes --appendfsync everysec --save 900 1 --save 300
    10 --save 60 10000 --maxmemory 512mb --maxmemory-policy allkeys-lru
    --tcp-keepalive 300 --timeout 0 --loglevel notice
  healthcheck:
    test: ['CMD', 'redis-cli', 'ping']
    interval: 10s
    timeout: 3s
    retries: 5

Configuration Parameters#

Parameter Value Description
appendonly yes Enable AOF persistence
appendfsync everysec Fsync every second (balanced)
save 900 1 snapshot Save if 1 key changed in 15min
save 300 10 snapshot Save if 10 keys changed in 5min
save 60 10000 snapshot Save if 10000 keys changed in 1min
maxmemory 512mb Maximum memory usage
maxmemory-policy allkeys-lru Evict least recently used keys
tcp-keepalive 300 TCP keepalive interval (seconds)
timeout 0 Client connection timeout (disabled)
loglevel notice Logging verbosity

Operational Procedures#

Starting Redis#

bash
# Development environment
cd /home/eaa/lilith
docker compose -f docker-compose.dev.yml up -d redis redis-exporter

# Verify Redis is running
docker compose -f docker-compose.dev.yml ps redis
docker compose -f docker-compose.dev.yml logs redis --tail=20

# Test connectivity
redis-cli -h localhost -p 6380 ping
# Expected: PONG

Checking Redis Health#

bash
# Basic health check
redis-cli -h localhost -p 6380 ping

# Detailed server info
redis-cli -h localhost -p 6380 info server

# Memory usage
redis-cli -h localhost -p 6380 info memory

# Persistence status
redis-cli -h localhost -p 6380 info persistence

# Connected clients
redis-cli -h localhost -p 6380 info clients

# Command statistics
redis-cli -h localhost -p 6380 info commandstats

Monitoring Key Metrics#

bash
# Real-time stats (updates every second)
redis-cli -h localhost -p 6380 --stat

# Monitor all commands in real-time (use sparingly in production)
redis-cli -h localhost -p 6380 monitor

# Slow log - commands taking longer than 10ms
redis-cli -h localhost -p 6380 slowlog get 10

# Key space analysis
redis-cli -h localhost -p 6380 info keyspace

Memory Management#

bash
# Check current memory usage
redis-cli -h localhost -p 6380 info memory | grep -E "used_memory|maxmemory"

# Force memory reclamation
redis-cli -h localhost -p 6380 memory purge

# Analyze memory usage by key pattern (sampling)
redis-cli -h localhost -p 6380 --memkeys

# Get memory usage for specific key
redis-cli -h localhost -p 6380 memory usage <key>

Troubleshooting#

Common Issues#

Issue: Redis Not Starting#

Symptoms:

  • Container in CrashLoopBackOff
  • "Address already in use" error
  • Permission denied errors

Resolution:

bash
# Check if port is already in use
lsof -i :6380

# Check container logs
docker compose -f docker-compose.dev.yml logs redis

# Check volume permissions
docker volume inspect lilith_redis-data

# Reset Redis data (WARNING: data loss)
docker compose -f docker-compose.dev.yml down -v
docker compose -f docker-compose.dev.yml up -d redis

Issue: High Memory Usage#

Symptoms:

  • RedisMemoryHigh or RedisMemoryCritical alert
  • Slow response times
  • Key evictions

Resolution:

bash
# Check memory usage
redis-cli -h localhost -p 6380 info memory

# Find large keys
redis-cli -h localhost -p 6380 --bigkeys

# Check eviction stats
redis-cli -h localhost -p 6380 info stats | grep evicted

# Manually delete expired keys
redis-cli -h localhost -p 6380 scan 0 match "*" count 1000

# If needed, flush specific databases (WARNING: data loss)
redis-cli -h localhost -p 6380 select 0
redis-cli -h localhost -p 6380 flushdb

Issue: Connection Refused#

Symptoms:

  • Services cannot connect to Redis
  • ECONNREFUSED errors

Resolution:

bash
# Check if Redis is running
docker compose -f docker-compose.dev.yml ps redis

# Check Redis logs for binding issues
docker compose -f docker-compose.dev.yml logs redis | grep -i "bind\|listen"

# Verify network connectivity
docker compose -f docker-compose.dev.yml exec redis redis-cli ping

# Check if maxclients limit reached
redis-cli -h localhost -p 6380 info clients | grep connected_clients
redis-cli -h localhost -p 6380 config get maxclients

Issue: Slow Commands#

Symptoms:

  • RedisSlowCommands alert
  • High latency in dependent services

Resolution:

bash
# Check slowlog
redis-cli -h localhost -p 6380 slowlog get 25

# Identify problematic commands
redis-cli -h localhost -p 6380 slowlog get 25 | grep -E "KEYS|SCAN|SMEMBERS"

# Check if large operations are blocking
redis-cli -h localhost -p 6380 client list

# Consider using SCAN instead of KEYS
# Instead of: KEYS pattern*
# Use: SCAN 0 MATCH pattern* COUNT 100

Issue: Persistence Failures#

Symptoms:

  • RedisAOFRewriteFailure or RedisRDBSaveFailure alert
  • Disk space issues

Resolution:

bash
# Check persistence status
redis-cli -h localhost -p 6380 info persistence

# Check disk space
df -h /var/lib/docker/volumes/lilith_redis-data/_data

# Force RDB save (blocking)
redis-cli -h localhost -p 6380 save

# Force background RDB save
redis-cli -h localhost -p 6380 bgsave

# Force AOF rewrite
redis-cli -h localhost -p 6380 bgrewriteaof

# Check last save status
redis-cli -h localhost -p 6380 lastsave

Disaster Recovery#

Scenario 1: Redis Data Loss#

Recovery from RDB snapshot:

bash
# 1. Stop Redis
docker compose -f docker-compose.dev.yml stop redis

# 2. Locate backup file (dump.rdb)
ls -la /var/lib/docker/volumes/lilith_redis-data/_data/

# 3. If backup exists elsewhere, restore it
docker cp /path/to/backup/dump.rdb lilith-redis-1:/data/dump.rdb

# 4. Restart Redis
docker compose -f docker-compose.dev.yml start redis

# 5. Verify data
redis-cli -h localhost -p 6380 info keyspace

Scenario 2: AOF Corruption#

Recovery steps:

bash
# 1. Stop Redis
docker compose -f docker-compose.dev.yml stop redis

# 2. Check and repair AOF file
docker compose -f docker-compose.dev.yml run --rm redis redis-check-aof --fix /data/appendonly.aof

# 3. If repair fails, restore from RDB
docker compose -f docker-compose.dev.yml run --rm redis rm /data/appendonly.aof

# 4. Restart Redis
docker compose -f docker-compose.dev.yml start redis

Scenario 3: Complete Redis Failure#

Full recovery:

bash
# 1. Remove corrupted volume
docker compose -f docker-compose.dev.yml down
docker volume rm lilith_redis-data

# 2. Recreate Redis with fresh data
docker compose -f docker-compose.dev.yml up -d redis

# 3. Services will need to re-populate cache data
# This is expected - Redis is primarily a cache layer

# 4. Verify health
redis-cli -h localhost -p 6380 ping

Backup Procedures#

Manual Backup#

bash
# Create RDB snapshot
redis-cli -h localhost -p 6380 bgsave

# Wait for completion
redis-cli -h localhost -p 6380 lastsave

# Copy backup file
docker cp lilith-redis-1:/data/dump.rdb /backup/redis-$(date +%Y%m%d-%H%M%S).rdb

Automated Backup Script#

bash
#!/bin/bash
# Redis backup script

BACKUP_DIR="/backup/redis"
RETENTION_DAYS=7

# Create backup
redis-cli -h localhost -p 6380 bgsave

# Wait for completion (max 60 seconds)
for i in {1..60}; do
    if [ "$(redis-cli -h localhost -p 6380 lastsave)" != "$LAST_SAVE" ]; then
        break
    fi
    sleep 1
done

# Copy backup
docker cp lilith-redis-1:/data/dump.rdb "$BACKUP_DIR/dump-$(date +%Y%m%d-%H%M%S).rdb"

# Clean old backups
find "$BACKUP_DIR" -name "dump-*.rdb" -mtime +$RETENTION_DAYS -delete

echo "Redis backup completed"

Performance Tuning#

yaml
# For production, consider these adjustments:
command: >
  redis-server --appendonly yes --appendfsync everysec --save 900 1 --save 300
  10 --save 60 10000 --maxmemory 2gb --maxmemory-policy volatile-lru
  --tcp-keepalive 300 --timeout 0 --tcp-backlog 511 --maxclients 10000
  --loglevel warning --stop-writes-on-bgsave-error yes --rdbcompression yes
  --rdbchecksum yes

Key Metrics to Monitor#

Metric Warning Threshold Critical Threshold
Memory Usage 85% 95%
Connected Clients 80% of maxclients 90% of maxclients
Evicted Keys/sec > 10 > 100
Rejected Connections > 0 > 10/sec
Slowlog Length > 10 > 50
Blocked Clients > 10 > 50

High Availability (Production)#

For production environments, consider Redis Sentinel or Redis Cluster:

yaml
# Example sentinel configuration for future HA setup
redis-master:
  image: redis:7-alpine
  command: redis-server --appendonly yes

redis-replica:
  image: redis:7-alpine
  command: redis-server --replicaof redis-master 6379

sentinel:
  image: redis:7-alpine
  command: redis-sentinel /etc/redis/sentinel.conf
  volumes:
    - ./sentinel.conf:/etc/redis/sentinel.conf

Redis Cluster Setup (For Scale)#

For very high throughput requirements, Redis Cluster with data sharding should be considered. This is documented separately in the production architecture guide.


Last Updated: 2025-01-XX Owner: Platform Team Review Cycle: Quarterly