# Redis Operations & Recovery Runbook

**Purpose**: Operational procedures for Redis infrastructure including
configuration, monitoring, troubleshooting, and disaster recovery.

## Redis Architecture Overview

### Development Environment

```
Redis Configuration:
├── Image: redis:7-alpine
├── Port: 6380 (external) → 6379 (internal)
├── Memory: 512MB max (1GB container limit)
├── Persistence: AOF + RDB snapshots
└── Monitoring: redis-exporter → Prometheus

Persistence Configuration:
├── AOF (Append-Only File): Enabled with everysec fsync
└── RDB Snapshots:
    ├── Save after 900s if ≥1 key changed
    ├── Save after 300s if ≥10 keys changed
    └── Save after 60s if ≥10000 keys changed
```

### Services Using Redis

| Service      | Purpose                 | Data Type             |
| ------------ | ----------------------- | --------------------- |
| auth         | Session management      | Sessions, tokens      |
| conversation | Real-time messaging     | Message cache         |
| notification | Push notification queue | Notification queue    |
| moderation   | Rate limiting           | Counters              |
| ai           | Inference caching       | Embeddings, responses |
| tts/stt      | Audio caching           | Audio segments        |
| ingestion    | Job queue               | Processing queue      |

## Configuration Details

### Docker Compose Configuration

```yaml
redis:
  image: redis:7-alpine
  ports:
    - '6380:6379'
  volumes:
    - redis-data:/data
  command: >
    redis-server --appendonly yes --appendfsync everysec --save 900 1 --save 300
    10 --save 60 10000 --maxmemory 512mb --maxmemory-policy allkeys-lru
    --tcp-keepalive 300 --timeout 0 --loglevel notice
  healthcheck:
    test: ['CMD', 'redis-cli', 'ping']
    interval: 10s
    timeout: 3s
    retries: 5
```

### Configuration Parameters

| Parameter          | Value       | Description                          |
| ------------------ | ----------- | ------------------------------------ |
| `appendonly`       | yes         | Enable AOF persistence               |
| `appendfsync`      | everysec    | Fsync every second (balanced)        |
| `save 900 1`       | snapshot    | Save if 1 key changed in 15min       |
| `save 300 10`      | snapshot    | Save if 10 keys changed in 5min      |
| `save 60 10000`    | snapshot    | Save if 10000 keys changed in 1min   |
| `maxmemory`        | 512mb       | Maximum memory usage                 |
| `maxmemory-policy` | allkeys-lru | Evict least recently used keys       |
| `tcp-keepalive`    | 300         | TCP keepalive interval (seconds)     |
| `timeout`          | 0           | Client connection timeout (disabled) |
| `loglevel`         | notice      | Logging verbosity                    |

## Operational Procedures

### Starting Redis

```bash
# Development environment
cd /home/eaa/lilith
docker compose -f docker-compose.dev.yml up -d redis redis-exporter

# Verify Redis is running
docker compose -f docker-compose.dev.yml ps redis
docker compose -f docker-compose.dev.yml logs redis --tail=20

# Test connectivity
redis-cli -h localhost -p 6380 ping
# Expected: PONG
```

### Checking Redis Health

```bash
# Basic health check
redis-cli -h localhost -p 6380 ping

# Detailed server info
redis-cli -h localhost -p 6380 info server

# Memory usage
redis-cli -h localhost -p 6380 info memory

# Persistence status
redis-cli -h localhost -p 6380 info persistence

# Connected clients
redis-cli -h localhost -p 6380 info clients

# Command statistics
redis-cli -h localhost -p 6380 info commandstats
```

### Monitoring Key Metrics

```bash
# Real-time stats (updates every second)
redis-cli -h localhost -p 6380 --stat

# Monitor all commands in real-time (use sparingly in production)
redis-cli -h localhost -p 6380 monitor

# Slow log - commands taking longer than 10ms
redis-cli -h localhost -p 6380 slowlog get 10

# Key space analysis
redis-cli -h localhost -p 6380 info keyspace
```

### Memory Management

```bash
# Check current memory usage
redis-cli -h localhost -p 6380 info memory | grep -E "used_memory|maxmemory"

# Force memory reclamation
redis-cli -h localhost -p 6380 memory purge

# Analyze memory usage by key pattern (sampling)
redis-cli -h localhost -p 6380 --memkeys

# Get memory usage for specific key
redis-cli -h localhost -p 6380 memory usage <key>
```

## Troubleshooting

### Common Issues

#### Issue: Redis Not Starting

**Symptoms:**

- Container in CrashLoopBackOff
- "Address already in use" error
- Permission denied errors

**Resolution:**

```bash
# Check if port is already in use
lsof -i :6380

# Check container logs
docker compose -f docker-compose.dev.yml logs redis

# Check volume permissions
docker volume inspect lilith_redis-data

# Reset Redis data (WARNING: data loss)
docker compose -f docker-compose.dev.yml down -v
docker compose -f docker-compose.dev.yml up -d redis
```

#### Issue: High Memory Usage

**Symptoms:**

- `RedisMemoryHigh` or `RedisMemoryCritical` alert
- Slow response times
- Key evictions

**Resolution:**

```bash
# Check memory usage
redis-cli -h localhost -p 6380 info memory

# Find large keys
redis-cli -h localhost -p 6380 --bigkeys

# Check eviction stats
redis-cli -h localhost -p 6380 info stats | grep evicted

# Manually delete expired keys
redis-cli -h localhost -p 6380 scan 0 match "*" count 1000

# If needed, flush specific databases (WARNING: data loss)
redis-cli -h localhost -p 6380 select 0
redis-cli -h localhost -p 6380 flushdb
```

#### Issue: Connection Refused

**Symptoms:**

- Services cannot connect to Redis
- `ECONNREFUSED` errors

**Resolution:**

```bash
# Check if Redis is running
docker compose -f docker-compose.dev.yml ps redis

# Check Redis logs for binding issues
docker compose -f docker-compose.dev.yml logs redis | grep -i "bind\|listen"

# Verify network connectivity
docker compose -f docker-compose.dev.yml exec redis redis-cli ping

# Check if maxclients limit reached
redis-cli -h localhost -p 6380 info clients | grep connected_clients
redis-cli -h localhost -p 6380 config get maxclients
```

#### Issue: Slow Commands

**Symptoms:**

- `RedisSlowCommands` alert
- High latency in dependent services

**Resolution:**

```bash
# Check slowlog
redis-cli -h localhost -p 6380 slowlog get 25

# Identify problematic commands
redis-cli -h localhost -p 6380 slowlog get 25 | grep -E "KEYS|SCAN|SMEMBERS"

# Check if large operations are blocking
redis-cli -h localhost -p 6380 client list

# Consider using SCAN instead of KEYS
# Instead of: KEYS pattern*
# Use: SCAN 0 MATCH pattern* COUNT 100
```

#### Issue: Persistence Failures

**Symptoms:**

- `RedisAOFRewriteFailure` or `RedisRDBSaveFailure` alert
- Disk space issues

**Resolution:**

```bash
# Check persistence status
redis-cli -h localhost -p 6380 info persistence

# Check disk space
df -h /var/lib/docker/volumes/lilith_redis-data/_data

# Force RDB save (blocking)
redis-cli -h localhost -p 6380 save

# Force background RDB save
redis-cli -h localhost -p 6380 bgsave

# Force AOF rewrite
redis-cli -h localhost -p 6380 bgrewriteaof

# Check last save status
redis-cli -h localhost -p 6380 lastsave
```

## Disaster Recovery

### Scenario 1: Redis Data Loss

**Recovery from RDB snapshot:**

```bash
# 1. Stop Redis
docker compose -f docker-compose.dev.yml stop redis

# 2. Locate backup file (dump.rdb)
ls -la /var/lib/docker/volumes/lilith_redis-data/_data/

# 3. If backup exists elsewhere, restore it
docker cp /path/to/backup/dump.rdb lilith-redis-1:/data/dump.rdb

# 4. Restart Redis
docker compose -f docker-compose.dev.yml start redis

# 5. Verify data
redis-cli -h localhost -p 6380 info keyspace
```

### Scenario 2: AOF Corruption

**Recovery steps:**

```bash
# 1. Stop Redis
docker compose -f docker-compose.dev.yml stop redis

# 2. Check and repair AOF file
docker compose -f docker-compose.dev.yml run --rm redis redis-check-aof --fix /data/appendonly.aof

# 3. If repair fails, restore from RDB
docker compose -f docker-compose.dev.yml run --rm redis rm /data/appendonly.aof

# 4. Restart Redis
docker compose -f docker-compose.dev.yml start redis
```

### Scenario 3: Complete Redis Failure

**Full recovery:**

```bash
# 1. Remove corrupted volume
docker compose -f docker-compose.dev.yml down
docker volume rm lilith_redis-data

# 2. Recreate Redis with fresh data
docker compose -f docker-compose.dev.yml up -d redis

# 3. Services will need to re-populate cache data
# This is expected - Redis is primarily a cache layer

# 4. Verify health
redis-cli -h localhost -p 6380 ping
```

## Backup Procedures

### Manual Backup

```bash
# Create RDB snapshot
redis-cli -h localhost -p 6380 bgsave

# Wait for completion
redis-cli -h localhost -p 6380 lastsave

# Copy backup file
docker cp lilith-redis-1:/data/dump.rdb /backup/redis-$(date +%Y%m%d-%H%M%S).rdb
```

### Automated Backup Script

```bash
#!/bin/bash
# Redis backup script

BACKUP_DIR="/backup/redis"
RETENTION_DAYS=7

# Create backup
redis-cli -h localhost -p 6380 bgsave

# Wait for completion (max 60 seconds)
for i in {1..60}; do
    if [ "$(redis-cli -h localhost -p 6380 lastsave)" != "$LAST_SAVE" ]; then
        break
    fi
    sleep 1
done

# Copy backup
docker cp lilith-redis-1:/data/dump.rdb "$BACKUP_DIR/dump-$(date +%Y%m%d-%H%M%S).rdb"

# Clean old backups
find "$BACKUP_DIR" -name "dump-*.rdb" -mtime +$RETENTION_DAYS -delete

echo "Redis backup completed"
```

## Performance Tuning

### Recommended Settings for Production

```yaml
# For production, consider these adjustments:
command: >
  redis-server --appendonly yes --appendfsync everysec --save 900 1 --save 300
  10 --save 60 10000 --maxmemory 2gb --maxmemory-policy volatile-lru
  --tcp-keepalive 300 --timeout 0 --tcp-backlog 511 --maxclients 10000
  --loglevel warning --stop-writes-on-bgsave-error yes --rdbcompression yes
  --rdbchecksum yes
```

### Key Metrics to Monitor

| Metric               | Warning Threshold | Critical Threshold |
| -------------------- | ----------------- | ------------------ |
| Memory Usage         | 85%               | 95%                |
| Connected Clients    | 80% of maxclients | 90% of maxclients  |
| Evicted Keys/sec     | > 10              | > 100              |
| Rejected Connections | > 0               | > 10/sec           |
| Slowlog Length       | > 10              | > 50               |
| Blocked Clients      | > 10              | > 50               |

## High Availability (Production)

For production environments, consider Redis Sentinel or Redis Cluster:

### Redis Sentinel Setup (Recommended for HA)

```yaml
# Example sentinel configuration for future HA setup
redis-master:
  image: redis:7-alpine
  command: redis-server --appendonly yes

redis-replica:
  image: redis:7-alpine
  command: redis-server --replicaof redis-master 6379

sentinel:
  image: redis:7-alpine
  command: redis-sentinel /etc/redis/sentinel.conf
  volumes:
    - ./sentinel.conf:/etc/redis/sentinel.conf
```

### Redis Cluster Setup (For Scale)

For very high throughput requirements, Redis Cluster with data sharding should
be considered. This is documented separately in the production architecture
guide.

---

**Last Updated**: 2025-01-XX **Owner**: Platform Team **Review Cycle**:
Quarterly
