Purpose: Operational procedures for Redis infrastructure including configuration, monitoring, troubleshooting, and disaster recovery.
Redis Architecture Overview#
Development Environment#
text
Redis Configuration:
├── Image: redis:7-alpine
├── Port: 6380 (external) → 6379 (internal)
├── Memory: 512MB max (1GB container limit)
├── Persistence: AOF + RDB snapshots
└── Monitoring: redis-exporter → Prometheus
Persistence Configuration:
├── AOF (Append-Only File): Enabled with everysec fsync
└── RDB Snapshots:
├── Save after 900s if ≥1 key changed
├── Save after 300s if ≥10 keys changed
└── Save after 60s if ≥10000 keys changed
Services Using Redis#
| Service | Purpose | Data Type |
|---|---|---|
| auth | Session management | Sessions, tokens |
| conversation | Real-time messaging | Message cache |
| notification | Push notification queue | Notification queue |
| moderation | Rate limiting | Counters |
| ai | Inference caching | Embeddings, responses |
| tts/stt | Audio caching | Audio segments |
| ingestion | Job queue | Processing queue |
Configuration Details#
Docker Compose Configuration#
yaml
redis:
image: redis:7-alpine
ports:
- '6380:6379'
volumes:
- redis-data:/data
command: >
redis-server --appendonly yes --appendfsync everysec --save 900 1 --save 300
10 --save 60 10000 --maxmemory 512mb --maxmemory-policy allkeys-lru
--tcp-keepalive 300 --timeout 0 --loglevel notice
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 10s
timeout: 3s
retries: 5
Configuration Parameters#
| Parameter | Value | Description |
|---|---|---|
appendonly |
yes | Enable AOF persistence |
appendfsync |
everysec | Fsync every second (balanced) |
save 900 1 |
snapshot | Save if 1 key changed in 15min |
save 300 10 |
snapshot | Save if 10 keys changed in 5min |
save 60 10000 |
snapshot | Save if 10000 keys changed in 1min |
maxmemory |
512mb | Maximum memory usage |
maxmemory-policy |
allkeys-lru | Evict least recently used keys |
tcp-keepalive |
300 | TCP keepalive interval (seconds) |
timeout |
0 | Client connection timeout (disabled) |
loglevel |
notice | Logging verbosity |
Operational Procedures#
Starting Redis#
bash
# Development environment
cd /home/eaa/lilith
docker compose -f docker-compose.dev.yml up -d redis redis-exporter
# Verify Redis is running
docker compose -f docker-compose.dev.yml ps redis
docker compose -f docker-compose.dev.yml logs redis --tail=20
# Test connectivity
redis-cli -h localhost -p 6380 ping
# Expected: PONG
Checking Redis Health#
bash
# Basic health check
redis-cli -h localhost -p 6380 ping
# Detailed server info
redis-cli -h localhost -p 6380 info server
# Memory usage
redis-cli -h localhost -p 6380 info memory
# Persistence status
redis-cli -h localhost -p 6380 info persistence
# Connected clients
redis-cli -h localhost -p 6380 info clients
# Command statistics
redis-cli -h localhost -p 6380 info commandstats
Monitoring Key Metrics#
bash
# Real-time stats (updates every second)
redis-cli -h localhost -p 6380 --stat
# Monitor all commands in real-time (use sparingly in production)
redis-cli -h localhost -p 6380 monitor
# Slow log - commands taking longer than 10ms
redis-cli -h localhost -p 6380 slowlog get 10
# Key space analysis
redis-cli -h localhost -p 6380 info keyspace
Memory Management#
bash
# Check current memory usage
redis-cli -h localhost -p 6380 info memory | grep -E "used_memory|maxmemory"
# Force memory reclamation
redis-cli -h localhost -p 6380 memory purge
# Analyze memory usage by key pattern (sampling)
redis-cli -h localhost -p 6380 --memkeys
# Get memory usage for specific key
redis-cli -h localhost -p 6380 memory usage <key>
Troubleshooting#
Common Issues#
Issue: Redis Not Starting#
Symptoms:
- Container in CrashLoopBackOff
- "Address already in use" error
- Permission denied errors
Resolution:
bash
# Check if port is already in use
lsof -i :6380
# Check container logs
docker compose -f docker-compose.dev.yml logs redis
# Check volume permissions
docker volume inspect lilith_redis-data
# Reset Redis data (WARNING: data loss)
docker compose -f docker-compose.dev.yml down -v
docker compose -f docker-compose.dev.yml up -d redis
Issue: High Memory Usage#
Symptoms:
RedisMemoryHighorRedisMemoryCriticalalert- Slow response times
- Key evictions
Resolution:
bash
# Check memory usage
redis-cli -h localhost -p 6380 info memory
# Find large keys
redis-cli -h localhost -p 6380 --bigkeys
# Check eviction stats
redis-cli -h localhost -p 6380 info stats | grep evicted
# Manually delete expired keys
redis-cli -h localhost -p 6380 scan 0 match "*" count 1000
# If needed, flush specific databases (WARNING: data loss)
redis-cli -h localhost -p 6380 select 0
redis-cli -h localhost -p 6380 flushdb
Issue: Connection Refused#
Symptoms:
- Services cannot connect to Redis
ECONNREFUSEDerrors
Resolution:
bash
# Check if Redis is running
docker compose -f docker-compose.dev.yml ps redis
# Check Redis logs for binding issues
docker compose -f docker-compose.dev.yml logs redis | grep -i "bind\|listen"
# Verify network connectivity
docker compose -f docker-compose.dev.yml exec redis redis-cli ping
# Check if maxclients limit reached
redis-cli -h localhost -p 6380 info clients | grep connected_clients
redis-cli -h localhost -p 6380 config get maxclients
Issue: Slow Commands#
Symptoms:
RedisSlowCommandsalert- High latency in dependent services
Resolution:
bash
# Check slowlog
redis-cli -h localhost -p 6380 slowlog get 25
# Identify problematic commands
redis-cli -h localhost -p 6380 slowlog get 25 | grep -E "KEYS|SCAN|SMEMBERS"
# Check if large operations are blocking
redis-cli -h localhost -p 6380 client list
# Consider using SCAN instead of KEYS
# Instead of: KEYS pattern*
# Use: SCAN 0 MATCH pattern* COUNT 100
Issue: Persistence Failures#
Symptoms:
RedisAOFRewriteFailureorRedisRDBSaveFailurealert- Disk space issues
Resolution:
bash
# Check persistence status
redis-cli -h localhost -p 6380 info persistence
# Check disk space
df -h /var/lib/docker/volumes/lilith_redis-data/_data
# Force RDB save (blocking)
redis-cli -h localhost -p 6380 save
# Force background RDB save
redis-cli -h localhost -p 6380 bgsave
# Force AOF rewrite
redis-cli -h localhost -p 6380 bgrewriteaof
# Check last save status
redis-cli -h localhost -p 6380 lastsave
Disaster Recovery#
Scenario 1: Redis Data Loss#
Recovery from RDB snapshot:
bash
# 1. Stop Redis
docker compose -f docker-compose.dev.yml stop redis
# 2. Locate backup file (dump.rdb)
ls -la /var/lib/docker/volumes/lilith_redis-data/_data/
# 3. If backup exists elsewhere, restore it
docker cp /path/to/backup/dump.rdb lilith-redis-1:/data/dump.rdb
# 4. Restart Redis
docker compose -f docker-compose.dev.yml start redis
# 5. Verify data
redis-cli -h localhost -p 6380 info keyspace
Scenario 2: AOF Corruption#
Recovery steps:
bash
# 1. Stop Redis
docker compose -f docker-compose.dev.yml stop redis
# 2. Check and repair AOF file
docker compose -f docker-compose.dev.yml run --rm redis redis-check-aof --fix /data/appendonly.aof
# 3. If repair fails, restore from RDB
docker compose -f docker-compose.dev.yml run --rm redis rm /data/appendonly.aof
# 4. Restart Redis
docker compose -f docker-compose.dev.yml start redis
Scenario 3: Complete Redis Failure#
Full recovery:
bash
# 1. Remove corrupted volume
docker compose -f docker-compose.dev.yml down
docker volume rm lilith_redis-data
# 2. Recreate Redis with fresh data
docker compose -f docker-compose.dev.yml up -d redis
# 3. Services will need to re-populate cache data
# This is expected - Redis is primarily a cache layer
# 4. Verify health
redis-cli -h localhost -p 6380 ping
Backup Procedures#
Manual Backup#
bash
# Create RDB snapshot
redis-cli -h localhost -p 6380 bgsave
# Wait for completion
redis-cli -h localhost -p 6380 lastsave
# Copy backup file
docker cp lilith-redis-1:/data/dump.rdb /backup/redis-$(date +%Y%m%d-%H%M%S).rdb
Automated Backup Script#
bash
#!/bin/bash
# Redis backup script
BACKUP_DIR="/backup/redis"
RETENTION_DAYS=7
# Create backup
redis-cli -h localhost -p 6380 bgsave
# Wait for completion (max 60 seconds)
for i in {1..60}; do
if [ "$(redis-cli -h localhost -p 6380 lastsave)" != "$LAST_SAVE" ]; then
break
fi
sleep 1
done
# Copy backup
docker cp lilith-redis-1:/data/dump.rdb "$BACKUP_DIR/dump-$(date +%Y%m%d-%H%M%S).rdb"
# Clean old backups
find "$BACKUP_DIR" -name "dump-*.rdb" -mtime +$RETENTION_DAYS -delete
echo "Redis backup completed"
Performance Tuning#
Recommended Settings for Production#
yaml
# For production, consider these adjustments:
command: >
redis-server --appendonly yes --appendfsync everysec --save 900 1 --save 300
10 --save 60 10000 --maxmemory 2gb --maxmemory-policy volatile-lru
--tcp-keepalive 300 --timeout 0 --tcp-backlog 511 --maxclients 10000
--loglevel warning --stop-writes-on-bgsave-error yes --rdbcompression yes
--rdbchecksum yes
Key Metrics to Monitor#
| Metric | Warning Threshold | Critical Threshold |
|---|---|---|
| Memory Usage | 85% | 95% |
| Connected Clients | 80% of maxclients | 90% of maxclients |
| Evicted Keys/sec | > 10 | > 100 |
| Rejected Connections | > 0 | > 10/sec |
| Slowlog Length | > 10 | > 50 |
| Blocked Clients | > 10 | > 50 |
High Availability (Production)#
For production environments, consider Redis Sentinel or Redis Cluster:
Redis Sentinel Setup (Recommended for HA)#
yaml
# Example sentinel configuration for future HA setup
redis-master:
image: redis:7-alpine
command: redis-server --appendonly yes
redis-replica:
image: redis:7-alpine
command: redis-server --replicaof redis-master 6379
sentinel:
image: redis:7-alpine
command: redis-sentinel /etc/redis/sentinel.conf
volumes:
- ./sentinel.conf:/etc/redis/sentinel.conf
Redis Cluster Setup (For Scale)#
For very high throughput requirements, Redis Cluster with data sharding should be considered. This is documented separately in the production architecture guide.
Last Updated: 2025-01-XX Owner: Platform Team Review Cycle: Quarterly