# Veritas Operations Runbook

Operational procedures for deploying, monitoring, and maintaining the Veritas AI
News Agency platform.

---

## Table of Contents

1. [Service Overview](#1-service-overview)
2. [Local Development Environment](#2-local-development-environment)
3. [Deployment Procedures](#3-deployment-procedures)
4. [Service Operations](#4-service-operations)
5. [Monitoring and Alerting](#5-monitoring-and-alerting)
6. [Incident Response](#6-incident-response)
7. [Database Operations](#7-database-operations)
8. [Queue and Worker Operations](#8-queue-and-worker-operations)
9. [Ingestion Service Operations](#9-ingestion-service-operations)
10. [Troubleshooting Guide](#10-troubleshooting-guide)
11. [Disaster Recovery](#11-disaster-recovery)
12. [Security Operations](#12-security-operations)
13. [Runbook Checklists](#13-runbook-checklists)

---

## 1. Service Overview

### 1.1 Service Inventory

| Service           | Type        | Port | Health Endpoint        | Description                         |
| ----------------- | ----------- | ---- | ---------------------- | ----------------------------------- |
| veritas-api       | HTTP API    | 3001 | `GET /health`          | Hono REST API serving all endpoints |
| veritas-web       | Web App     | 3000 | `GET /health`          | Next.js 15 PWA frontend             |
| veritas-ingestion | CLI Workers | N/A  | N/A (process-based)    | RSS/sitemap collection and pipeline |
| PostgreSQL        | Database    | 5432 | `pg_isready`           | Primary data store with pgvector    |
| Redis             | Cache/Queue | 6379 | `redis-cli ping`       | Rate limiting, BullMQ job queues    |
| Elasticsearch     | Search      | 9200 | `GET /_cluster/health` | Article full-text search            |

### 1.2 Service Dependencies

```
veritas-web ──→ veritas-api ──→ PostgreSQL
                             ──→ Redis
                             ──→ Elasticsearch

veritas-ingestion (scheduler) ──→ PostgreSQL
                              ──→ External RSS feeds

veritas-ingestion (producer)  ──→ PostgreSQL
                              ──→ Redis (BullMQ)

veritas-ingestion (worker)    ──→ PostgreSQL
                              ──→ Redis (BullMQ)
```

### 1.3 Required Environment Variables

#### API Service

| Variable                       | Required | Default            | Description                         |
| ------------------------------ | -------- | ------------------ | ----------------------------------- |
| `PORT`                         | No       | `3001`             | HTTP listen port                    |
| `HOST`                         | No       | `0.0.0.0`          | HTTP listen host                    |
| `DATABASE_URL`                 | Yes      | —                  | PostgreSQL connection string        |
| `REDIS_URL`                    | No       | —                  | Redis connection string             |
| `ELASTICSEARCH_URL`            | No       | —                  | Elasticsearch connection string     |
| `ELASTICSEARCH_ARTICLES_INDEX` | No       | `veritas-articles` | Elasticsearch index name            |
| `VERITAS_ACCESS_TOKEN_SECRET`  | Yes      | —                  | JWT signing secret (HMAC-SHA256)    |
| `NEXTAUTH_SECRET`              | No       | —                  | Fallback JWT secret                 |
| `TIPS_ENCRYPTION_KEY`          | Yes      | —                  | AES-256-GCM encryption key for tips |
| `LOG_LEVEL`                    | No       | `info`             | Logging verbosity                   |
| `CORS_ENABLED`                 | No       | `false`            | Enable CORS middleware              |
| `CORS_ORIGINS`                 | No       | `*`                | Comma-separated allowed origins     |
| `RATE_LIMIT_ENABLED`           | No       | `false`            | Enable sliding-window rate limiting |
| `RATE_LIMIT_MAX`               | No       | `120`              | Max requests per window             |
| `RATE_LIMIT_WINDOW_SECONDS`    | No       | `60`               | Rate limit window in seconds        |
| `RATE_LIMIT_PREFIX`            | No       | `rl:`              | Redis key prefix for rate limiting  |
| `API_KEY_AUTH_ENABLED`         | No       | `false`            | Enable API key authentication       |
| `API_KEYS`                     | No       | —                  | Comma-separated valid API keys      |
| `API_KEY_HEADER`               | No       | `x-api-key`        | Header name for API key             |
| `OPENAPI_ENABLED`              | No       | `true`             | Enable OpenAPI docs at `/docs`      |

#### Ingestion Service

| Variable                       | Required | Default                                               | Description                           |
| ------------------------------ | -------- | ----------------------------------------------------- | ------------------------------------- |
| `VERITAS_DATABASE_URL`         | No       | `postgresql://oshun:oshun_dev@localhost:5432/veritas` | PostgreSQL connection string          |
| `VERITAS_REDIS_URL`            | No       | `redis://localhost:6379`                              | Redis for BullMQ queues               |
| `VERITAS_ONCE`                 | No       | `false`                                               | Run single tick then exit             |
| `VERITAS_POLL_INTERVAL_MS`     | No       | `60000`                                               | Scheduler/producer loop interval      |
| `VERITAS_DISABLE_RSS`          | No       | `false`                                               | Disable RSS collection                |
| `VERITAS_RSS_LIMIT`            | No       | `100`                                                 | Max feeds per scheduler tick          |
| `VERITAS_RSS_CONCURRENCY`      | No       | `10`                                                  | Parallel feed fetches                 |
| `VERITAS_RSS_TIMEOUT_MS`       | No       | `30000`                                               | Per-feed fetch timeout                |
| `VERITAS_RSS_MAX_BYTES`        | No       | `10485760`                                            | Max feed response size (10 MB)        |
| `VERITAS_USER_AGENT`           | No       | `VeritasBot/1.0 (+https://veritas.news)`              | HTTP User-Agent string                |
| `VERITAS_ENABLE_SITEMAPS`      | No       | `false`                                               | Enable XML sitemap collection         |
| `VERITAS_SITEMAP_LIMIT`        | No       | `10`                                                  | Max sitemaps per tick                 |
| `VERITAS_SITEMAP_CONCURRENCY`  | No       | `5`                                                   | Parallel sitemap fetches              |
| `VERITAS_SITEMAP_TIMEOUT_MS`   | No       | `60000`                                               | Per-sitemap fetch timeout             |
| `VERITAS_SITEMAP_MAX_BYTES`    | No       | `52428800`                                            | Max sitemap response size (50 MB)     |
| `VERITAS_SITEMAP_SINCE_DAYS`   | No       | `7`                                                   | Only process entries from last N days |
| `VERITAS_SITEMAP_MAX_CHILDREN` | No       | `50`                                                  | Max child sitemaps to process         |
| `VERITAS_ENABLE_SOCIAL`        | No       | `false`                                               | Enable social media collection        |
| `VERITAS_SOCIAL_BEARER_TOKEN`  | No       | —                                                     | Social media API bearer token         |
| `VERITAS_PRODUCER_LIMIT`       | No       | `100`                                                 | Max items enqueued per tick           |
| `VERITAS_WORKER_CONCURRENCY`   | No       | `4`                                                   | Parallel job processing               |
| `VERITAS_TIMEOUT_MS`           | No       | `30000`                                               | Per-job processing timeout            |
| `VERITAS_MAX_BYTES`            | No       | `10485760`                                            | Max content size for scraping         |
| `VERITAS_SCRAPE_MISSING`       | No       | `true`                                                | Scrape if no HTML in feed item        |

---

## 2. Local Development Environment

### 2.1 Prerequisites

- Node.js 20+
- pnpm 10+
- Docker and Docker Compose
- Access to the Oshun monorepo

### 2.2 Starting Infrastructure

```bash
# Start core infrastructure (PostgreSQL, Redis, MinIO, Mailpit)
docker compose -f docker/docker-compose.dev.yml up -d

# Verify services are healthy
docker compose -f docker/docker-compose.dev.yml ps

# Install dependencies
pnpm install
```

### 2.3 Starting Services

```bash
# Start the API server (development mode with hot reload)
pnpm nx serve veritas-api

# Start the web frontend
pnpm nx serve veritas-web

# Start ingestion scheduler
pnpm nx run veritas-ingestion:scheduler

# Start ingestion producer
pnpm nx run veritas-ingestion:producer

# Start ingestion worker
pnpm nx run veritas-ingestion:worker

# Seed the database with Ghana news sources
pnpm nx run veritas-ingestion:seed
```

### 2.4 Optional Infrastructure Profiles

```bash
# Elasticsearch for article search
docker compose -f docker/docker-compose.dev.yml --profile search up -d

# Qdrant for vector similarity
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Kafka for streaming
docker compose -f docker/docker-compose.dev.yml --profile streaming up -d

# Prometheus + Grafana + Jaeger for observability
docker compose -f docker/docker-compose.dev.yml --profile observability up -d

# PgAdmin, Redis Commander, Kafka UI
docker compose -f docker/docker-compose.dev.yml --profile tools up -d

# Everything
docker compose -f docker/docker-compose.dev.yml --profile all up -d
```

### 2.5 Verifying Local Setup

```bash
# Check API health
curl http://localhost:3001/health

# Check readiness (includes dependency checks)
curl http://localhost:3001/readyz

# Check detailed health with latencies
curl http://localhost:3001/health/detailed

# Verify database connectivity
docker exec oshun-dev-postgres pg_isready -U oshun

# Verify Redis
docker exec oshun-dev-redis redis-cli ping
```

---

## 3. Deployment Procedures

### 3.1 Deployment Environments

| Environment | Trigger                  | Strategy                         | Approval       |
| ----------- | ------------------------ | -------------------------------- | -------------- |
| Staging     | Push to `main`           | Rolling deployment               | Automatic      |
| Production  | Tag `v*.*.*` or release  | Canary (10%) → Blue-green → Full | Manual release |
| Hotfix      | Manual workflow dispatch | Direct rolling or canary         | Manual trigger |

### 3.2 CI/CD Pipeline Overview

The deployment pipeline is defined in `.github/workflows/deploy.yml` (583 lines)
and consists of the following stages:

```
Push to main
  │
  ├─→ Plan (determine affected apps, environment, strategy)
  │
  ├─→ Build Images (matrix per affected app)
  │     ├── Docker build with Buildx
  │     ├── Multi-platform: linux/amd64, linux/arm64
  │     ├── Trivy vulnerability scan (CRITICAL/HIGH)
  │     └── Push to ghcr.io
  │
  ├─→ Deploy Staging
  │     ├── Helm upgrade --install
  │     ├── Smoke tests (/health endpoints)
  │     └── 10-minute timeout
  │
  ├─→ Deploy Canary (production only, 10% traffic)
  │     ├── Canary deployment with Ingress weights
  │     ├── 5-minute soak validation
  │     └── Multiple health checks
  │
  ├─→ Deploy Production (full rollout)
  │     ├── Blue-green: switch via service selector
  │     │   OR
  │     ├── Rolling: standard Helm upgrade
  │     └── Post-deployment verification (health + perf < 2s)
  │
  └─→ Rollback (on failure)
        ├── kubectl rollout undo deployment
        └── Slack notification
```

### 3.3 Docker Image Build

All services use multi-stage Docker builds:

| Dockerfile        | Base Image         | Use Case                  | Health Check               |
| ----------------- | ------------------ | ------------------------- | -------------------------- |
| Dockerfile.node   | node:22-alpine     | API and Node.js services  | `GET /health` on port 3000 |
| Dockerfile.web    | Nginx              | SPA/PWA web applications  | `GET /health` on port 8080 |
| Dockerfile.worker | node:22-alpine     | Background job processors | SIGUSR1 signal             |
| Dockerfile.gpu    | CUDA 12.2 + Ubuntu | ML/AI workers             | CUDA availability check    |

Build characteristics:

- Non-root user (UID 1001: `oshun`)
- `dumb-init` for proper signal handling
- pnpm frozen lockfile for deterministic builds
- Source maps enabled for debugging
- Prisma client generation when applicable

### 3.4 Staging Deployment

```bash
# Automatic on push to main, or manual:
gh workflow run deploy.yml \
  -f environment=staging \
  -f strategy=rolling

# Verify staging deployment
curl https://staging.veritas.oshun.io/health
curl https://staging.veritas.oshun.io/readyz
```

### 3.5 Production Deployment

```bash
# Option 1: Create a release tag
git tag v1.2.3
git push origin v1.2.3

# Option 2: Create a GitHub release
gh release create v1.2.3 --title "v1.2.3" --notes "Release notes"

# Option 3: Manual dispatch with canary
gh workflow run deploy.yml \
  -f environment=production \
  -f strategy=canary
```

### 3.6 Production Deployment Verification

After a production deployment, verify:

```bash
# 1. Health check
curl https://veritas.oshun.io/health
# Expected: {"status":"ok","service":"veritas-api","timestamp":"..."}

# 2. Readiness check (dependency verification)
curl https://veritas.oshun.io/readyz
# Expected: {"status":"ok","checks":{"database":"ok","redis":"ok","elasticsearch":"ok"}}

# 3. Performance check (response time < 2s)
time curl -o /dev/null -s -w '%{time_total}' https://veritas.oshun.io/api/v1/articles?limit=5

# 4. API key authentication (if enabled)
curl -H "x-api-key: $API_KEY" https://veritas.oshun.io/api/v1/articles

# 5. Check rate limit headers
curl -I https://veritas.oshun.io/api/v1/articles
# Look for: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset
```

### 3.7 Rollback Procedures

#### Automatic Rollback

The CI/CD pipeline automatically rolls back on deployment failure:

```
deploy-production failure → rollback job triggers
  → kubectl rollout undo deployment
  → Slack notification sent
```

#### Manual Rollback

```bash
# Rollback to previous Helm release
helm rollback veritas-api -n oshun-production-veritas

# Or rollback Kubernetes deployment directly
kubectl rollout undo deployment/veritas-api -n oshun-production-veritas

# Verify rollback
kubectl rollout status deployment/veritas-api -n oshun-production-veritas
curl https://veritas.oshun.io/health
```

#### Emergency Rollback (Docker)

```bash
# List available image tags
docker images ghcr.io/greychimp/veritas-api --format "{{.Tag}}"

# Deploy specific previous version
helm upgrade veritas-api infrastructure/helm/oshun-service \
  --set image.tag=v1.2.2 \
  -n oshun-production-veritas \
  -f infrastructure/helm/values/production.yaml
```

---

## 4. Service Operations

### 4.1 API Service Startup Sequence

The API service (`apps/veritas/api/src/main.ts`) initializes in this order:

1. Read environment configuration
2. Initialize PostgreSQL connection pool
3. Initialize Redis client
4. Initialize Elasticsearch client
5. Create Hono application with middleware stack
6. Start HTTP server on configured port
7. Log startup information

**Middleware stack** (applied in order):

1. Request logging
2. Request timing (`X-Response-Time` header)
3. Security headers
4. CORS (if enabled)
5. Rate limiting on `/api/*` (if enabled, requires Redis)
6. API key authentication on `/api/*` (if enabled)
7. Global error handler

### 4.2 API Service Graceful Shutdown

The API handles `SIGTERM` and `SIGINT` signals:

1. Close HTTP server (stop accepting new connections)
2. Close PostgreSQL connection pool
3. Close Redis connection (`quit()` with `disconnect()` fallback)
4. Close Elasticsearch client
5. Exit with code 0 (success) or 1 (error during shutdown)

Unhandled promise rejections and uncaught exceptions trigger the shutdown
sequence automatically.

### 4.3 Health Check Endpoints

| Endpoint               | Purpose             | Response Codes | Use Case                   |
| ---------------------- | ------------------- | -------------- | -------------------------- |
| `GET /health`          | Basic liveness      | 200            | Load balancer health check |
| `GET /healthz`         | K8s liveness probe  | 200            | Kubernetes liveness probe  |
| `GET /readyz`          | K8s readiness probe | 200, 503       | Kubernetes readiness probe |
| `GET /health/detailed` | Full diagnostics    | 200            | Monitoring dashboards      |

#### Readiness Probe Response Statuses

| Status      | Meaning                       | HTTP Code |
| ----------- | ----------------------------- | --------- |
| `ok`        | All dependencies healthy      | 200       |
| `degraded`  | Some dependencies unavailable | 200       |
| `unhealthy` | All dependencies unavailable  | 503       |

The readiness probe checks: database connectivity, Redis connectivity, and
Elasticsearch cluster health.

### 4.4 Scaling Configuration

#### Horizontal Pod Autoscaler (Kubernetes)

```yaml
# API service autoscaling
minReplicas: 2
maxReplicas: 10
metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 70
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 80
```

#### Connection Pool Sizing

| Service          | Pool Size | Idle Timeout | Connect Timeout |
| ---------------- | --------- | ------------ | --------------- |
| API (PostgreSQL) | 20        | 30s          | 10s             |
| Ingestion        | 10        | 30s          | 10s             |

#### Redis Connection Settings

- `maxRetriesPerRequest: null` (unlimited retries for BullMQ reliability)
- `enableReadyCheck: false` (performance optimization)
- Exponential backoff retry: `delay = min(attempts * 1000, 30000)`

---

## 5. Monitoring and Alerting

### 5.1 Metrics Collection

**Prometheus** scrapes metrics at 15-second intervals in production (30s in
development):

| Target        | Endpoint         | Labels                   |
| ------------- | ---------------- | ------------------------ |
| veritas-api   | `:3001/metrics`  | `domain: veritas`        |
| PostgreSQL    | Exporter `:9187` | `service: postgres`      |
| Redis         | Exporter `:9121` | `service: redis`         |
| Elasticsearch | Exporter `:9114` | `service: elasticsearch` |
| Traefik       | `:8082/metrics`  | `service: traefik`       |

### 5.2 Key Metrics to Monitor

#### API Service

| Metric                       | Warning Threshold | Critical Threshold | Action                         |
| ---------------------------- | ----------------- | ------------------ | ------------------------------ |
| HTTP response time (p95)     | > 500ms           | > 2000ms           | Scale up or investigate DB     |
| HTTP error rate (5xx)        | > 1%              | > 5%               | Check logs, rollback if needed |
| Rate limit rejections (429s) | > 50/min          | > 200/min          | Review rate limit config       |
| Active connections           | > 80% of max      | > 95% of max       | Scale up replicas              |

#### Database

| Metric                      | Warning Threshold | Critical Threshold | Action                      |
| --------------------------- | ----------------- | ------------------ | --------------------------- |
| Connection pool utilization | > 70%             | > 90%              | Increase pool size or scale |
| Query latency (p95)         | > 100ms           | > 500ms            | Analyze slow queries        |
| Disk usage                  | > 70%             | > 85%              | Expand storage or archive   |
| Replication lag             | > 5s              | > 30s              | Check replica health        |
| Dead tuples ratio           | > 10%             | > 20%              | Run VACUUM ANALYZE          |

#### Redis

| Metric               | Warning Threshold | Critical Threshold | Action                      |
| -------------------- | ----------------- | ------------------ | --------------------------- |
| Memory usage         | > 70%             | > 85%              | Review eviction policies    |
| Connected clients    | > 80% of max      | > 95% of max       | Scale or review connections |
| Queue depth (BullMQ) | > 1000 jobs       | > 5000 jobs        | Scale workers               |
| Failed jobs (BullMQ) | > 10/hour         | > 50/hour          | Investigate failures        |

#### Ingestion Pipeline

| Metric                         | Warning Threshold | Critical Threshold | Action                       |
| ------------------------------ | ----------------- | ------------------ | ---------------------------- |
| Feed consecutive failures      | > 5               | > 10               | Check feed URL, disable feed |
| Items processed per minute     | < 10              | < 1                | Check worker health          |
| Unprocessed feed items backlog | > 500             | > 2000             | Scale workers                |
| Scheduler tick duration        | > 120s            | > 300s             | Reduce RSS concurrency       |

### 5.3 Grafana Dashboards

Access Grafana at `http://localhost:3100` (development) or the production
Grafana URL.

Default credentials: `admin`/`admin` (change in production).

Dashboards are auto-provisioned from `/var/lib/grafana/dashboards`.

### 5.4 Distributed Tracing

Jaeger provides distributed tracing:

- **UI**: `http://localhost:16686` (development)
- **Collector**: port 14268
- **Agent**: port 6831/UDP
- **Zipkin compatible**: port 9411

Use traces to investigate:

- Cross-service latency
- Database query patterns
- External API call chains
- Error propagation paths

### 5.5 Log Analysis

All services emit structured JSON logs. Key fields:

```json
{
  "at": "2025-01-26T12:00:00.000Z",
  "event": "event_name",
  "level": "info|warn|error",
  "message": "Human-readable description",
  "error": "Error message (if applicable)",
  "stack": "Stack trace (development mode only)"
}
```

#### Ingestion Service Log Events

| Event                     | Level | Meaning                                |
| ------------------------- | ----- | -------------------------------------- |
| `scheduler_tick_complete` | info  | Scheduler finished a collection cycle  |
| `producer_tick_complete`  | info  | Producer finished enqueuing jobs       |
| `job_completed`           | info  | Worker successfully processed an item  |
| `job_failed`              | error | Worker failed to process an item       |
| `worker_error`            | error | Worker encountered an unexpected error |
| `feed_fetch_failed`       | warn  | Individual feed fetch failed           |
| `scrape_failed`           | warn  | Content scraping failed for an item    |

#### Useful Log Queries

```bash
# Find all errors in the last hour
journalctl -u veritas-api --since "1 hour ago" | jq 'select(.level == "error")'

# Find rate limit violations
journalctl -u veritas-api --since "1 hour ago" | jq 'select(.status == 429)'

# Find ingestion failures
journalctl -u veritas-ingestion --since "1 hour ago" | jq 'select(.event == "job_failed")'

# Count events by type
journalctl -u veritas-ingestion --since "1 hour ago" | jq -r '.event' | sort | uniq -c | sort -rn
```

---

## 6. Incident Response

### 6.1 Severity Levels

| Severity | Description                                     | Response Time     | Escalation          |
| -------- | ----------------------------------------------- | ----------------- | ------------------- |
| SEV-1    | Complete service outage, data loss risk         | Immediate         | All hands on deck   |
| SEV-2    | Major functionality degraded, user-facing       | 15 minutes        | On-call + team lead |
| SEV-3    | Minor functionality degraded, workaround exists | 1 hour            | On-call engineer    |
| SEV-4    | Non-urgent issue, cosmetic, low impact          | Next business day | Ticket in backlog   |

### 6.2 Incident Response Procedures

#### SEV-1: Complete Service Outage

1. **Acknowledge** the incident in the alerting channel
2. **Assess** scope:

   ```bash
   # Check all health endpoints
   curl https://veritas.oshun.io/health
   curl https://veritas.oshun.io/readyz
   curl https://veritas.oshun.io/health/detailed

   # Check Kubernetes pod status
   kubectl get pods -n oshun-production-veritas
   kubectl describe pod <pod-name> -n oshun-production-veritas

   # Check recent events
   kubectl get events -n oshun-production-veritas --sort-by='.lastTimestamp'
   ```

3. **Mitigate** immediately:
   - If deployment-related: rollback to previous version
   - If infrastructure-related: check database, Redis, Elasticsearch status
   - If traffic-related: enable emergency rate limiting
4. **Communicate** status to stakeholders
5. **Resolve** the root cause
6. **Post-mortem** within 48 hours

#### SEV-2: Degraded Performance

1. **Identify** the degraded component via `/health/detailed`
2. **Check** recent deployments and changes
3. **Scale** if resource-constrained:
   ```bash
   kubectl scale deployment veritas-api --replicas=5 -n oshun-production-veritas
   ```
4. **Investigate** logs and metrics for the affected component
5. **Resolve** or apply temporary mitigation

### 6.3 Common Incident Scenarios

#### API Returns 503 (Service Unavailable)

**Symptoms**: `/readyz` returns `unhealthy`, users see 503 errors.

**Diagnosis**:

```bash
# Check which dependency is down
curl https://veritas.oshun.io/readyz
# Response shows which checks failed: database, redis, elasticsearch

# Check database
kubectl exec -it $(kubectl get pod -l app=postgres -n oshun-production-veritas -o name) \
  -- pg_isready -U oshun

# Check Redis
kubectl exec -it $(kubectl get pod -l app=redis -n oshun-production-veritas -o name) \
  -- redis-cli ping

# Check Elasticsearch
curl http://elasticsearch:9200/_cluster/health
```

**Resolution**:

- Database down: Check pod status, disk space, connection limits
- Redis down: Check memory, restart if OOM killed
- Elasticsearch down: Check cluster health, disk watermarks

#### High Error Rate (5xx Responses)

**Symptoms**: Error rate exceeds 5%, alerts firing.

**Diagnosis**:

```bash
# Check API logs for error patterns
kubectl logs -l app=veritas-api -n oshun-production-veritas --tail=100 | \
  jq 'select(.level == "error")'

# Check for OOM kills
kubectl get events -n oshun-production-veritas | grep OOMKilled

# Check resource usage
kubectl top pods -n oshun-production-veritas
```

**Resolution**:

- OOM kills: Increase memory limits
- Database errors: Check connection pool exhaustion
- Unhandled exceptions: Deploy hotfix or rollback

#### Ingestion Pipeline Stalled

**Symptoms**: No new articles appearing, unprocessed backlog growing.

**Diagnosis**:

```bash
# Check unprocessed items count
psql $VERITAS_DATABASE_URL -c "
  SELECT COUNT(*) as unprocessed
  FROM feed_items fi
  WHERE NOT EXISTS (
    SELECT 1 FROM normalized_feed_items nfi
    WHERE nfi.feed_item_id = fi.id
  );
"

# Check BullMQ queue depth
redis-cli LLEN bull:veritas:content-processing:wait
redis-cli LLEN bull:veritas:content-processing:active
redis-cli ZCARD bull:veritas:content-processing:failed

# Check scheduler logs
kubectl logs -l app=veritas-scheduler -n oshun-production-veritas --tail=50

# Check worker logs
kubectl logs -l app=veritas-worker -n oshun-production-veritas --tail=50
```

**Resolution**:

- Scheduler stopped: Restart scheduler process
- Workers stopped: Restart worker processes
- Queue full of failed jobs: Investigate failure reasons, retry or clear
- All feeds failing: Check network connectivity, DNS resolution

#### Rate Limiting Too Aggressive

**Symptoms**: Legitimate users receiving 429 responses.

**Diagnosis**:

```bash
# Check current rate limit configuration
echo "RATE_LIMIT_MAX=$RATE_LIMIT_MAX RATE_LIMIT_WINDOW_SECONDS=$RATE_LIMIT_WINDOW_SECONDS"

# Check Redis for rate limit keys
redis-cli KEYS "rl:*" | head -20
redis-cli ZCARD "rl:<client-ip>"
```

**Resolution**:

- Increase `RATE_LIMIT_MAX` or `RATE_LIMIT_WINDOW_SECONDS`
- Whitelist specific API keys from rate limiting
- Scale API replicas to distribute load

---

## 7. Database Operations

### 7.1 Connection Management

PostgreSQL is configured with multiple databases for domain isolation:

| Database    | Purpose                   |
| ----------- | ------------------------- |
| `oshun_dev` | Main development database |
| `veritas`   | Veritas production data   |

Extensions enabled on all databases:

- `uuid-ossp` — UUID generation
- `pgcrypto` — Cryptographic functions
- `vector` — pgvector for embeddings
- `pg_trgm` — Trigram text similarity

Schemas per database: `public`, `app`, `analytics`, `staging`

### 7.2 Database Connection Strings

```bash
# Development
postgresql://oshun:oshun_dev@localhost:5432/oshun_dev

# Veritas-specific
postgresql://oshun:oshun_dev@localhost:5432/veritas
```

### 7.3 Backup Procedures

#### RDS Automated Backups (AWS)

| Environment | Retention | Multi-AZ | Deletion Protection |
| ----------- | --------- | -------- | ------------------- |
| Production  | 30 days   | Yes      | Yes                 |
| Staging     | 7 days    | No       | No                  |

#### Manual Backup

```bash
# Full database dump
pg_dump $VERITAS_DATABASE_URL -Fc -f veritas_backup_$(date +%Y%m%d_%H%M%S).dump

# Schema-only backup
pg_dump $VERITAS_DATABASE_URL --schema-only -f veritas_schema_$(date +%Y%m%d).sql

# Data-only backup for specific table
pg_dump $VERITAS_DATABASE_URL -t feed_items --data-only -f feed_items_backup.sql
```

#### Restore from Backup

```bash
# Restore full database
pg_restore -d $VERITAS_DATABASE_URL -c veritas_backup_20250126_120000.dump

# Restore specific table
pg_restore -d $VERITAS_DATABASE_URL -t feed_items veritas_backup.dump
```

### 7.4 Database Maintenance

#### Routine Maintenance Queries

```sql
-- Check table sizes
SELECT schemaname, tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as total_size,
  pg_size_pretty(pg_relation_size(schemaname||'.'||tablename)) as table_size,
  pg_size_pretty(pg_indexes_size(schemaname||'.'||tablename::regclass)) as index_size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 20;

-- Check dead tuples (need VACUUM if high)
SELECT schemaname, relname, n_dead_tup, n_live_tup,
  round(n_dead_tup::numeric / NULLIF(n_live_tup, 0) * 100, 2) as dead_pct,
  last_vacuum, last_autovacuum
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC
LIMIT 20;

-- Check active connections
SELECT datname, count(*) as connections,
  count(*) FILTER (WHERE state = 'active') as active,
  count(*) FILTER (WHERE state = 'idle') as idle,
  count(*) FILTER (WHERE state = 'idle in transaction') as idle_in_txn
FROM pg_stat_activity
GROUP BY datname;

-- Check long-running queries
SELECT pid, now() - pg_stat_activity.query_start AS duration,
  query, state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes'
  AND state != 'idle'
ORDER BY duration DESC;

-- Check index usage
SELECT schemaname, tablename, indexname,
  idx_scan, idx_tup_read, idx_tup_fetch,
  pg_size_pretty(pg_relation_size(indexrelid)) as index_size
FROM pg_stat_user_indexes
ORDER BY idx_scan ASC
LIMIT 20;
```

#### Periodic Maintenance Tasks

```bash
# Analyze all tables (update query planner statistics)
psql $VERITAS_DATABASE_URL -c "ANALYZE;"

# Vacuum analyze specific table
psql $VERITAS_DATABASE_URL -c "VACUUM ANALYZE feed_items;"

# Reindex bloated indexes
psql $VERITAS_DATABASE_URL -c "REINDEX TABLE CONCURRENTLY feed_items;"
```

### 7.5 Database Schema Initialization

The PostgreSQL initialization script (`scripts/init-multiple-databases.sh`)
automatically:

1. Creates databases from `POSTGRES_MULTIPLE_DATABASES` env var
2. Enables extensions: `uuid-ossp`, `pgcrypto`, `vector`, `pg_trgm`
3. Creates schemas: `app`, `analytics`, `staging`
4. Sets schema permissions and default privileges
5. Installs vector search helper functions:
   - `vector_cosine_similarity(vector, vector)` → float
   - `vector_l2_distance(vector, vector)` → float
   - `suggest_vector_index()` → index creation SQL

---

## 8. Queue and Worker Operations

### 8.1 BullMQ Queue Architecture

| Queue                        | Purpose                   | Consumer          |
| ---------------------------- | ------------------------- | ----------------- |
| `veritas:content-processing` | Feed item processing jobs | Ingestion workers |

### 8.2 Job Configuration

| Setting                 | Value               | Description                   |
| ----------------------- | ------------------- | ----------------------------- |
| Job type                | `process_feed_item` | Processes a single feed item  |
| Max attempts            | 3                   | Total attempts before failure |
| Backoff strategy        | Exponential         | 5000ms base delay             |
| Completed job retention | 24 hours            | Auto-removed after 24h        |
| Completed job limit     | 10,000              | Max completed jobs in Redis   |
| Failed job retention    | 7 days              | Failed jobs kept for analysis |
| Job ID format           | `process:{uuid}`    | Prevents duplicate enqueuing  |

### 8.3 Queue Monitoring Commands

```bash
# Check queue sizes
redis-cli LLEN bull:veritas:content-processing:wait      # Waiting jobs
redis-cli LLEN bull:veritas:content-processing:active     # In-progress jobs
redis-cli ZCARD bull:veritas:content-processing:delayed   # Delayed/retry jobs
redis-cli ZCARD bull:veritas:content-processing:failed    # Failed jobs
redis-cli ZCARD bull:veritas:content-processing:completed # Completed jobs

# Get failed job details
redis-cli ZRANGE bull:veritas:content-processing:failed 0 9 WITHSCORES

# Get specific job data
redis-cli HGETALL bull:veritas:content-processing:<job-id>

# Check worker activity
redis-cli SMEMBERS bull:veritas:content-processing:workers
```

### 8.4 Queue Recovery Operations

#### Clear Failed Jobs

```bash
# Remove all failed jobs
redis-cli DEL bull:veritas:content-processing:failed

# Retry specific failed job
# (requires application-level retry logic or BullMQ dashboard)
```

#### Drain Queue

```bash
# Remove all waiting jobs (emergency only)
redis-cli DEL bull:veritas:content-processing:wait

# Remove all delayed jobs
redis-cli DEL bull:veritas:content-processing:delayed
```

#### Reset Queue Completely

```bash
# WARNING: This removes all queue state
redis-cli KEYS "bull:veritas:content-processing:*" | xargs redis-cli DEL
```

---

## 9. Ingestion Service Operations

### 9.1 Process Architecture

The ingestion service runs as three independent processes:

```
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Scheduler  │     │   Producer   │     │    Worker     │
│              │     │              │     │              │
│ Fetches RSS  │────→│ Enqueues     │────→│ Processes    │
│ and sitemaps │ DB  │ unprocessed  │Queue│ feed items   │
│              │     │ items        │     │              │
└──────────────┘     └──────────────┘     └──────────────┘
       │                    │                    │
       ▼                    ▼                    ▼
   PostgreSQL         Redis (BullMQ)        PostgreSQL
```

### 9.2 Starting and Stopping

```bash
# Start individual processes
node dist/apps/veritas/ingestion/main.js scheduler
node dist/apps/veritas/ingestion/main.js producer
node dist/apps/veritas/ingestion/main.js worker

# Run single tick (useful for debugging)
VERITAS_ONCE=true node dist/apps/veritas/ingestion/main.js scheduler
VERITAS_ONCE=true node dist/apps/veritas/ingestion/main.js producer

# Seed database with Ghana news sources
node dist/apps/veritas/ingestion/main.js seed

# Dry-run seed (preview without database changes)
node dist/apps/veritas/ingestion/main.js seed --dry-run

# Seed summary only (no database connection needed)
node dist/apps/veritas/ingestion/main.js seed --summary
```

All processes handle `SIGTERM` and `SIGINT` for graceful shutdown, closing:

1. Active workers/intervals
2. Redis connections
3. Database connection pools

### 9.3 Processing Pipeline Stages

Each feed item goes through six stages:

| Stage                  | Purpose                                  | Output Table                                                     |
| ---------------------- | ---------------------------------------- | ---------------------------------------------------------------- |
| 1. Normalization       | Extract clean text from HTML/scrape      | `normalized_feed_items`                                          |
| 2. Deduplication       | Simhash64 fingerprinting + LSH bucketing | `content_fingerprints`, `fingerprint_buckets`, `dedupe_mappings` |
| 3. Language Detection  | Detect language from first 120 chars     | `language_detections`                                            |
| 4. Timestamp Normalize | Resolve publication time with timezone   | `timestamp_normalizations`                                       |
| 5. Media Extraction    | Extract images, videos, audio from HTML  | `media_assets`                                                   |
| 6. Source Attribution  | Link content to original sources         | `source_attributions`                                            |

### 9.4 Feed Management

#### Check Feed Health

```sql
-- Feeds with high failure rates
SELECT s.name as source_name, sf.url, sf.consecutive_failures,
  sf.last_success_at, sf.last_fetched_at, sf.status
FROM source_feeds sf
JOIN sources s ON s.id = sf.source_id
WHERE sf.consecutive_failures > 3
ORDER BY sf.consecutive_failures DESC;

-- Feed collection summary
SELECT sf.status, COUNT(*) as count,
  AVG(sf.consecutive_failures) as avg_failures,
  MAX(sf.last_fetched_at) as most_recent_fetch
FROM source_feeds sf
GROUP BY sf.status;

-- Feeds due for collection
SELECT COUNT(*) as due_feeds
FROM source_feeds
WHERE status = 'active' AND next_fetch_at <= NOW();
```

#### Disable/Enable Feeds

```sql
-- Disable a problematic feed
UPDATE source_feeds
SET status = 'inactive'
WHERE url = 'https://example.com/rss';

-- Re-enable a feed and reset failures
UPDATE source_feeds
SET status = 'active', consecutive_failures = 0,
    next_fetch_at = NOW()
WHERE url = 'https://example.com/rss';

-- Suspend all feeds from a source
UPDATE source_feeds
SET status = 'suspended'
WHERE source_id = (SELECT id FROM sources WHERE canonical_domain = 'example.com');
```

### 9.5 Deduplication System

The ingestion pipeline uses Simhash64 with LSH (Locality-Sensitive Hashing) for
content deduplication:

- **Fingerprinting**: 64-bit Simhash of normalized text
- **LSH Buckets**: 4 bands for approximate nearest-neighbor lookup
- **Match Criteria**: Min length ratio 0.6, max Hamming distance 3 bits
- **Canonical Mapping**: Duplicate items map to a canonical item ID

#### Check Deduplication Stats

```sql
-- Deduplication rate
SELECT
  COUNT(*) as total_items,
  COUNT(DISTINCT dm.canonical_normalized_feed_item_id) as unique_items,
  COUNT(*) - COUNT(DISTINCT dm.canonical_normalized_feed_item_id) as duplicates,
  ROUND((COUNT(*) - COUNT(DISTINCT dm.canonical_normalized_feed_item_id))::numeric
    / COUNT(*) * 100, 2) as dedup_pct
FROM dedupe_mappings dm;

-- Items with most duplicates
SELECT dm.canonical_normalized_feed_item_id, COUNT(*) as duplicate_count,
  nfi.title
FROM dedupe_mappings dm
JOIN normalized_feed_items nfi ON nfi.id = dm.canonical_normalized_feed_item_id
GROUP BY dm.canonical_normalized_feed_item_id, nfi.title
ORDER BY duplicate_count DESC
LIMIT 20;
```

### 9.6 Backoff and Rate Limiting

The scheduler implements adaptive backoff for failed feeds:

- **Exponential backoff**: `2^failures * 60000ms` (capped)
- **Retry-After respect**: Honors HTTP 429/503 `Retry-After` headers
- **Conditional requests**: Uses `ETag` and `Last-Modified` for 304 optimization
- **EWMA latency tracking**: Adapts polling intervals based on response times

---

## 10. Troubleshooting Guide

### 10.1 API Issues

#### API Not Starting

```bash
# Check if port is in use
lsof -i :3001

# Check environment variables
env | grep -E "^(PORT|HOST|DATABASE_URL|REDIS_URL)"

# Test database connectivity directly
psql $DATABASE_URL -c "SELECT 1;"

# Test Redis connectivity
redis-cli -u $REDIS_URL ping

# Check Node.js version
node --version  # Should be 22+
```

#### Slow API Responses

```bash
# Check detailed health (includes latency info)
curl https://veritas.oshun.io/health/detailed

# Check database query performance
psql $DATABASE_URL -c "
  SELECT query, calls, mean_exec_time, total_exec_time
  FROM pg_stat_statements
  ORDER BY mean_exec_time DESC
  LIMIT 10;
"

# Check connection pool status
psql $DATABASE_URL -c "
  SELECT count(*), state
  FROM pg_stat_activity
  WHERE datname = 'veritas'
  GROUP BY state;
"
```

#### JWT Authentication Failures

```bash
# Verify token secret is set
echo $VERITAS_ACCESS_TOKEN_SECRET | wc -c  # Should be > 0

# Decode a JWT (header and payload only)
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq .

# Check token expiration
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq '.exp | todate'
```

### 10.2 Ingestion Issues

#### Feeds Not Being Collected

```bash
# Check scheduler is running
ps aux | grep "veritas.*scheduler"

# Run a single tick manually for debugging
VERITAS_ONCE=true VERITAS_RSS_LIMIT=5 \
  node dist/apps/veritas/ingestion/main.js scheduler

# Check feed due dates
psql $VERITAS_DATABASE_URL -c "
  SELECT url, status, next_fetch_at, consecutive_failures
  FROM source_feeds
  WHERE status = 'active'
  ORDER BY next_fetch_at
  LIMIT 10;
"
```

#### Items Not Being Processed

```bash
# Check unprocessed count
psql $VERITAS_DATABASE_URL -c "
  SELECT COUNT(*) as unprocessed
  FROM feed_items fi
  WHERE NOT EXISTS (
    SELECT 1 FROM normalized_feed_items nfi
    WHERE nfi.feed_item_id = fi.id
  );
"

# Check queue status
redis-cli LLEN bull:veritas:content-processing:wait
redis-cli LLEN bull:veritas:content-processing:active

# Check worker is running
ps aux | grep "veritas.*worker"

# Run worker with verbose logging
LOG_LEVEL=debug node dist/apps/veritas/ingestion/main.js worker
```

#### Content Scraping Failures

```bash
# Test scraping a specific URL
VERITAS_ONCE=true VERITAS_SCRAPE_MISSING=true \
  node dist/apps/veritas/ingestion/main.js worker

# Check scrape results
psql $VERITAS_DATABASE_URL -c "
  SELECT fi.url, nfi.title, nfi.content_text IS NOT NULL as has_text,
    nfi.content_html IS NOT NULL as has_html
  FROM feed_items fi
  LEFT JOIN normalized_feed_items nfi ON nfi.feed_item_id = fi.id
  ORDER BY fi.created_at DESC
  LIMIT 20;
"
```

### 10.3 Infrastructure Issues

#### PostgreSQL Connection Issues

```bash
# Check max connections
psql $DATABASE_URL -c "SHOW max_connections;"

# Check current connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# Kill long-running idle connections
psql $DATABASE_URL -c "
  SELECT pg_terminate_backend(pid)
  FROM pg_stat_activity
  WHERE state = 'idle'
  AND query_start < NOW() - interval '30 minutes'
  AND datname = 'veritas';
"
```

#### Redis Memory Issues

```bash
# Check Redis memory usage
redis-cli INFO memory | grep used_memory_human

# Check key count by pattern
redis-cli --scan --pattern "rl:*" | wc -l   # Rate limit keys
redis-cli --scan --pattern "bull:*" | wc -l  # BullMQ keys

# Clear rate limit data (if needed)
redis-cli --scan --pattern "rl:*" | xargs redis-cli DEL

# Check eviction policy
redis-cli CONFIG GET maxmemory-policy
```

#### Elasticsearch Issues

```bash
# Check cluster health
curl http://localhost:9200/_cluster/health?pretty

# Check index status
curl http://localhost:9200/_cat/indices?v

# Check disk watermarks
curl http://localhost:9200/_cluster/settings?include_defaults | \
  jq '.defaults.cluster.routing.allocation.disk'
```

---

## 11. Disaster Recovery

### 11.1 Recovery Point Objective (RPO) and Recovery Time Objective (RTO)

| Component     | RPO          | RTO        | Backup Method            |
| ------------- | ------------ | ---------- | ------------------------ |
| PostgreSQL    | 24 hours     | 1 hour     | RDS automated + manual   |
| Redis         | Ephemeral    | 15 minutes | Restart and repopulate   |
| Elasticsearch | Rebuild      | 2 hours    | Re-index from PostgreSQL |
| Feed items    | 0 (re-fetch) | Variable   | Re-collect from sources  |
| Configuration | 0 (in Git)   | 15 minutes | Redeploy from repository |

### 11.2 Database Recovery

```bash
# Restore from RDS automated backup (AWS Console or CLI)
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier veritas-prod \
  --target-db-instance-identifier veritas-restored \
  --restore-time "2025-01-26T10:00:00Z"

# Restore from manual dump
pg_restore -d $NEW_DATABASE_URL -c veritas_backup.dump
```

### 11.3 Full Environment Recovery

1. **Infrastructure**: Apply Terraform configuration

   ```bash
   cd infrastructure/terraform
   terraform init
   terraform plan -var-file=environments/production.tfvars
   terraform apply -var-file=environments/production.tfvars
   ```

2. **Database**: Restore from backup or RDS snapshot

3. **Application**: Deploy from latest stable tag

   ```bash
   gh workflow run deploy.yml \
     -f environment=production \
     -f strategy=rolling
   ```

4. **Ingestion**: Seed sources and restart collection

   ```bash
   node dist/apps/veritas/ingestion/main.js seed
   # Scheduler will automatically begin collecting from next_fetch_at
   ```

5. **Verification**: Run full health checks
   ```bash
   curl https://veritas.oshun.io/readyz
   curl https://veritas.oshun.io/health/detailed
   ```

---

## 12. Security Operations

### 12.1 Secret Rotation

#### JWT Secret Rotation

1. Generate new secret:
   ```bash
   openssl rand -base64 64
   ```
2. Update `VERITAS_ACCESS_TOKEN_SECRET` in environment configuration
3. Deploy the change (rolling deployment)
4. Existing tokens signed with the old secret will fail validation — users must
   re-authenticate

#### API Key Rotation

1. Generate new API key:
   ```bash
   openssl rand -hex 32
   ```
2. Add new key to `API_KEYS` (comma-separated list supports multiple keys)
3. Deploy the change
4. Notify API consumers to switch to the new key
5. Remove old key from `API_KEYS` after migration period

#### Tips Encryption Key Rotation

The `TIPS_ENCRYPTION_KEY` is used for AES-256-GCM encryption of anonymous tips.
Rotation requires re-encrypting existing tips:

1. Generate new key:
   ```bash
   openssl rand -hex 16  # 32 hex chars = 16 bytes = 128 bits
   ```
2. Write a migration script to decrypt with old key and re-encrypt with new key
3. Update environment variable and deploy
4. Verify decryption works with test tips

### 12.2 TLS Certificate Management

Traefik handles TLS certificate management automatically via Let's Encrypt:

- **Challenge type**: HTTP-01 (port 80)
- **Storage**: `/etc/traefik/acme/acme.json`
- **Auto-renewal**: Handled by Traefik

To force certificate renewal:

```bash
# Remove existing certificates (Traefik will request new ones)
rm /etc/traefik/acme/acme.json
# Restart Traefik
docker restart traefik
```

### 12.3 Security Headers

The API sets the following security headers on all responses:

- `X-Frame-Options: SAMEORIGIN`
- `X-Content-Type-Options: nosniff`
- `X-XSS-Protection: 1; mode=block`
- `Referrer-Policy: strict-origin-when-cross-origin`
- `Permissions-Policy: camera=(), microphone=(), geolocation=()`

The Nginx web server adds equivalent headers for the frontend.

### 12.4 Rate Limiting Configuration

The API uses sliding-window rate limiting with Redis sorted sets:

| Setting      | Default   | Description                        |
| ------------ | --------- | ---------------------------------- |
| Window       | 60s       | Sliding window duration            |
| Max requests | 120       | Maximum requests per window        |
| Redis prefix | `rl:`     | Key prefix for rate limit counters |
| Fallback     | In-memory | Used when Redis is unavailable     |

Response headers:

- `X-RateLimit-Limit` — Maximum requests allowed
- `X-RateLimit-Remaining` — Requests remaining in current window
- `X-RateLimit-Reset` — Unix timestamp when window resets

---

## 13. Runbook Checklists

### 13.1 Pre-Deployment Checklist

- [ ] All CI checks pass (build, lint, test)
- [ ] Trivy vulnerability scan shows no CRITICAL/HIGH issues
- [ ] Database migrations (if any) are backward compatible
- [ ] Environment variables are configured in target environment
- [ ] Rollback plan documented and tested
- [ ] Monitoring alerts are configured
- [ ] Team notified of upcoming deployment

### 13.2 Post-Deployment Checklist

- [ ] Health endpoint returns `200 OK`
- [ ] Readiness endpoint returns `ok` for all dependencies
- [ ] API response time < 2s at p95
- [ ] No increase in error rate (5xx responses)
- [ ] Rate limiting headers present in responses
- [ ] Ingestion pipeline processing items normally
- [ ] BullMQ queue depth is stable or decreasing
- [ ] No new entries in error logs
- [ ] Slack deployment notification received

### 13.3 Weekly Maintenance Checklist

- [ ] Review database table sizes and growth trends
- [ ] Check dead tuple ratios and VACUUM status
- [ ] Review feed health (consecutive failures)
- [ ] Check Redis memory usage and eviction rate
- [ ] Review Elasticsearch index sizes and shard health
- [ ] Audit BullMQ failed job count and reasons
- [ ] Review API error rate trends
- [ ] Check TLS certificate expiration dates
- [ ] Review and rotate stale API keys

### 13.4 Monthly Maintenance Checklist

- [ ] Review and update rate limit thresholds
- [ ] Analyze slow query logs and optimize
- [ ] Review database backup integrity (test restore)
- [ ] Audit user access and API key usage
- [ ] Review security scan results
- [ ] Update dependencies (minor/patch versions)
- [ ] Review monitoring alert thresholds
- [ ] Capacity planning review (storage, compute, connections)

### 13.5 Incident Post-Mortem Template

```markdown
## Incident Report: [TITLE]

**Date**: YYYY-MM-DD **Duration**: HH:MM **Severity**: SEV-1/2/3/4 **Author**:
[Name]

### Summary

[One paragraph describing the incident]

### Timeline

- HH:MM — [Event]
- HH:MM — [Event]

### Root Cause

[Detailed technical explanation]

### Impact

- Users affected: [number/percentage]
- Duration of impact: [time]
- Data loss: [yes/no, details]

### Resolution

[Steps taken to resolve]

### Action Items

- [ ] [Action] — Owner — Due date
- [ ] [Action] — Owner — Due date

### Lessons Learned

[What went well, what could be improved]
```
