Eve library

Veritas Operations Runbook

Architecture

Revision 1

On this page

Operational procedures for deploying, monitoring, and maintaining the Veritas AI News Agency platform.


Table of Contents#

  1. Service Overview
  2. Local Development Environment
  3. Deployment Procedures
  4. Service Operations
  5. Monitoring and Alerting
  6. Incident Response
  7. Database Operations
  8. Queue and Worker Operations
  9. Ingestion Service Operations
  10. Troubleshooting Guide
  11. Disaster Recovery
  12. Security Operations
  13. Runbook Checklists

1. Service Overview#

1.1 Service Inventory#

Service Type Port Health Endpoint Description
veritas-api HTTP API 3001 GET /health Hono REST API serving all endpoints
veritas-web Web App 3000 GET /health Next.js 15 PWA frontend
veritas-ingestion CLI Workers N/A N/A (process-based) RSS/sitemap collection and pipeline
PostgreSQL Database 5432 pg_isready Primary data store with pgvector
Redis Cache/Queue 6379 redis-cli ping Rate limiting, BullMQ job queues
Elasticsearch Search 9200 GET /_cluster/health Article full-text search

1.2 Service Dependencies#

text
veritas-web ──→ veritas-api ──→ PostgreSQL
                             ──→ Redis
                             ──→ Elasticsearch

veritas-ingestion (scheduler) ──→ PostgreSQL
                              ──→ External RSS feeds

veritas-ingestion (producer)  ──→ PostgreSQL
                              ──→ Redis (BullMQ)

veritas-ingestion (worker)    ──→ PostgreSQL
                              ──→ Redis (BullMQ)

1.3 Required Environment Variables#

API Service#

Variable Required Default Description
PORT No 3001 HTTP listen port
HOST No 0.0.0.0 HTTP listen host
DATABASE_URL Yes — PostgreSQL connection string
REDIS_URL No — Redis connection string
ELASTICSEARCH_URL No — Elasticsearch connection string
ELASTICSEARCH_ARTICLES_INDEX No veritas-articles Elasticsearch index name
VERITAS_ACCESS_TOKEN_SECRET Yes — JWT signing secret (HMAC-SHA256)
NEXTAUTH_SECRET No — Fallback JWT secret
TIPS_ENCRYPTION_KEY Yes — AES-256-GCM encryption key for tips
LOG_LEVEL No info Logging verbosity
CORS_ENABLED No false Enable CORS middleware
CORS_ORIGINS No * Comma-separated allowed origins
RATE_LIMIT_ENABLED No false Enable sliding-window rate limiting
RATE_LIMIT_MAX No 120 Max requests per window
RATE_LIMIT_WINDOW_SECONDS No 60 Rate limit window in seconds
RATE_LIMIT_PREFIX No rl: Redis key prefix for rate limiting
API_KEY_AUTH_ENABLED No false Enable API key authentication
API_KEYS No — Comma-separated valid API keys
API_KEY_HEADER No x-api-key Header name for API key
OPENAPI_ENABLED No true Enable OpenAPI docs at /docs

Ingestion Service#

Variable Required Default Description
VERITAS_DATABASE_URL No postgresql://oshun:oshun_dev@localhost:5432/veritas PostgreSQL connection string
VERITAS_REDIS_URL No redis://localhost:6379 Redis for BullMQ queues
VERITAS_ONCE No false Run single tick then exit
VERITAS_POLL_INTERVAL_MS No 60000 Scheduler/producer loop interval
VERITAS_DISABLE_RSS No false Disable RSS collection
VERITAS_RSS_LIMIT No 100 Max feeds per scheduler tick
VERITAS_RSS_CONCURRENCY No 10 Parallel feed fetches
VERITAS_RSS_TIMEOUT_MS No 30000 Per-feed fetch timeout
VERITAS_RSS_MAX_BYTES No 10485760 Max feed response size (10 MB)
VERITAS_USER_AGENT No VeritasBot/1.0 (+https://veritas.news) HTTP User-Agent string
VERITAS_ENABLE_SITEMAPS No false Enable XML sitemap collection
VERITAS_SITEMAP_LIMIT No 10 Max sitemaps per tick
VERITAS_SITEMAP_CONCURRENCY No 5 Parallel sitemap fetches
VERITAS_SITEMAP_TIMEOUT_MS No 60000 Per-sitemap fetch timeout
VERITAS_SITEMAP_MAX_BYTES No 52428800 Max sitemap response size (50 MB)
VERITAS_SITEMAP_SINCE_DAYS No 7 Only process entries from last N days
VERITAS_SITEMAP_MAX_CHILDREN No 50 Max child sitemaps to process
VERITAS_ENABLE_SOCIAL No false Enable social media collection
VERITAS_SOCIAL_BEARER_TOKEN No — Social media API bearer token
VERITAS_PRODUCER_LIMIT No 100 Max items enqueued per tick
VERITAS_WORKER_CONCURRENCY No 4 Parallel job processing
VERITAS_TIMEOUT_MS No 30000 Per-job processing timeout
VERITAS_MAX_BYTES No 10485760 Max content size for scraping
VERITAS_SCRAPE_MISSING No true Scrape if no HTML in feed item

2. Local Development Environment#

2.1 Prerequisites#

  • Node.js 20+
  • pnpm 10+
  • Docker and Docker Compose
  • Access to the Oshun monorepo

2.2 Starting Infrastructure#

bash
# Start core infrastructure (PostgreSQL, Redis, MinIO, Mailpit)
docker compose -f docker/docker-compose.dev.yml up -d

# Verify services are healthy
docker compose -f docker/docker-compose.dev.yml ps

# Install dependencies
pnpm install

2.3 Starting Services#

bash
# Start the API server (development mode with hot reload)
pnpm nx serve veritas-api

# Start the web frontend
pnpm nx serve veritas-web

# Start ingestion scheduler
pnpm nx run veritas-ingestion:scheduler

# Start ingestion producer
pnpm nx run veritas-ingestion:producer

# Start ingestion worker
pnpm nx run veritas-ingestion:worker

# Seed the database with Ghana news sources
pnpm nx run veritas-ingestion:seed

2.4 Optional Infrastructure Profiles#

bash
# Elasticsearch for article search
docker compose -f docker/docker-compose.dev.yml --profile search up -d

# Qdrant for vector similarity
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Kafka for streaming
docker compose -f docker/docker-compose.dev.yml --profile streaming up -d

# Prometheus + Grafana + Jaeger for observability
docker compose -f docker/docker-compose.dev.yml --profile observability up -d

# PgAdmin, Redis Commander, Kafka UI
docker compose -f docker/docker-compose.dev.yml --profile tools up -d

# Everything
docker compose -f docker/docker-compose.dev.yml --profile all up -d

2.5 Verifying Local Setup#

bash
# Check API health
curl http://localhost:3001/health

# Check readiness (includes dependency checks)
curl http://localhost:3001/readyz

# Check detailed health with latencies
curl http://localhost:3001/health/detailed

# Verify database connectivity
docker exec oshun-dev-postgres pg_isready -U oshun

# Verify Redis
docker exec oshun-dev-redis redis-cli ping

3. Deployment Procedures#

3.1 Deployment Environments#

Environment Trigger Strategy Approval
Staging Push to main Rolling deployment Automatic
Production Tag v*.*.* or release Canary (10%) → Blue-green → Full Manual release
Hotfix Manual workflow dispatch Direct rolling or canary Manual trigger

3.2 CI/CD Pipeline Overview#

The deployment pipeline is defined in .github/workflows/deploy.yml (583 lines) and consists of the following stages:

text
Push to main
  │
  ├─→ Plan (determine affected apps, environment, strategy)
  │
  ├─→ Build Images (matrix per affected app)
  │     ├── Docker build with Buildx
  │     ├── Multi-platform: linux/amd64, linux/arm64
  │     ├── Trivy vulnerability scan (CRITICAL/HIGH)
  │     └── Push to ghcr.io
  │
  ├─→ Deploy Staging
  │     ├── Helm upgrade --install
  │     ├── Smoke tests (/health endpoints)
  │     └── 10-minute timeout
  │
  ├─→ Deploy Canary (production only, 10% traffic)
  │     ├── Canary deployment with Ingress weights
  │     ├── 5-minute soak validation
  │     └── Multiple health checks
  │
  ├─→ Deploy Production (full rollout)
  │     ├── Blue-green: switch via service selector
  │     │   OR
  │     ├── Rolling: standard Helm upgrade
  │     └── Post-deployment verification (health + perf < 2s)
  │
  └─→ Rollback (on failure)
        ├── kubectl rollout undo deployment
        └── Slack notification

3.3 Docker Image Build#

All services use multi-stage Docker builds:

Dockerfile Base Image Use Case Health Check
Dockerfile.node node:22-alpine API and Node.js services GET /health on port 3000
Dockerfile.web Nginx SPA/PWA web applications GET /health on port 8080
Dockerfile.worker node:22-alpine Background job processors SIGUSR1 signal
Dockerfile.gpu CUDA 12.2 + Ubuntu ML/AI workers CUDA availability check

Build characteristics:

  • Non-root user (UID 1001: oshun)
  • dumb-init for proper signal handling
  • pnpm frozen lockfile for deterministic builds
  • Source maps enabled for debugging
  • Prisma client generation when applicable

3.4 Staging Deployment#

bash
# Automatic on push to main, or manual:
gh workflow run deploy.yml \
  -f environment=staging \
  -f strategy=rolling

# Verify staging deployment
curl https://staging.veritas.oshun.io/health
curl https://staging.veritas.oshun.io/readyz

3.5 Production Deployment#

bash
# Option 1: Create a release tag
git tag v1.2.3
git push origin v1.2.3

# Option 2: Create a GitHub release
gh release create v1.2.3 --title "v1.2.3" --notes "Release notes"

# Option 3: Manual dispatch with canary
gh workflow run deploy.yml \
  -f environment=production \
  -f strategy=canary

3.6 Production Deployment Verification#

After a production deployment, verify:

bash
# 1. Health check
curl https://veritas.oshun.io/health
# Expected: {"status":"ok","service":"veritas-api","timestamp":"..."}

# 2. Readiness check (dependency verification)
curl https://veritas.oshun.io/readyz
# Expected: {"status":"ok","checks":{"database":"ok","redis":"ok","elasticsearch":"ok"}}

# 3. Performance check (response time < 2s)
time curl -o /dev/null -s -w '%{time_total}' https://veritas.oshun.io/api/v1/articles?limit=5

# 4. API key authentication (if enabled)
curl -H "x-api-key: $API_KEY" https://veritas.oshun.io/api/v1/articles

# 5. Check rate limit headers
curl -I https://veritas.oshun.io/api/v1/articles
# Look for: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset

3.7 Rollback Procedures#

Automatic Rollback#

The CI/CD pipeline automatically rolls back on deployment failure:

text
deploy-production failure → rollback job triggers
  → kubectl rollout undo deployment
  → Slack notification sent

Manual Rollback#

bash
# Rollback to previous Helm release
helm rollback veritas-api -n oshun-production-veritas

# Or rollback Kubernetes deployment directly
kubectl rollout undo deployment/veritas-api -n oshun-production-veritas

# Verify rollback
kubectl rollout status deployment/veritas-api -n oshun-production-veritas
curl https://veritas.oshun.io/health

Emergency Rollback (Docker)#

bash
# List available image tags
docker images ghcr.io/greychimp/veritas-api --format "{{.Tag}}"

# Deploy specific previous version
helm upgrade veritas-api infrastructure/helm/oshun-service \
  --set image.tag=v1.2.2 \
  -n oshun-production-veritas \
  -f infrastructure/helm/values/production.yaml

4. Service Operations#

4.1 API Service Startup Sequence#

The API service (apps/veritas/api/src/main.ts) initializes in this order:

  1. Read environment configuration
  2. Initialize PostgreSQL connection pool
  3. Initialize Redis client
  4. Initialize Elasticsearch client
  5. Create Hono application with middleware stack
  6. Start HTTP server on configured port
  7. Log startup information

Middleware stack (applied in order):

  1. Request logging
  2. Request timing (X-Response-Time header)
  3. Security headers
  4. CORS (if enabled)
  5. Rate limiting on /api/* (if enabled, requires Redis)
  6. API key authentication on /api/* (if enabled)
  7. Global error handler

4.2 API Service Graceful Shutdown#

The API handles SIGTERM and SIGINT signals:

  1. Close HTTP server (stop accepting new connections)
  2. Close PostgreSQL connection pool
  3. Close Redis connection (quit() with disconnect() fallback)
  4. Close Elasticsearch client
  5. Exit with code 0 (success) or 1 (error during shutdown)

Unhandled promise rejections and uncaught exceptions trigger the shutdown sequence automatically.

4.3 Health Check Endpoints#

Endpoint Purpose Response Codes Use Case
GET /health Basic liveness 200 Load balancer health check
GET /healthz K8s liveness probe 200 Kubernetes liveness probe
GET /readyz K8s readiness probe 200, 503 Kubernetes readiness probe
GET /health/detailed Full diagnostics 200 Monitoring dashboards

Readiness Probe Response Statuses#

Status Meaning HTTP Code
ok All dependencies healthy 200
degraded Some dependencies unavailable 200
unhealthy All dependencies unavailable 503

The readiness probe checks: database connectivity, Redis connectivity, and Elasticsearch cluster health.

4.4 Scaling Configuration#

Horizontal Pod Autoscaler (Kubernetes)#

yaml
# API service autoscaling
minReplicas: 2
maxReplicas: 10
metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 70
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 80

Connection Pool Sizing#

Service Pool Size Idle Timeout Connect Timeout
API (PostgreSQL) 20 30s 10s
Ingestion 10 30s 10s

Redis Connection Settings#

  • maxRetriesPerRequest: null (unlimited retries for BullMQ reliability)
  • enableReadyCheck: false (performance optimization)
  • Exponential backoff retry: delay = min(attempts * 1000, 30000)

5. Monitoring and Alerting#

5.1 Metrics Collection#

Prometheus scrapes metrics at 15-second intervals in production (30s in development):

Target Endpoint Labels
veritas-api :3001/metrics domain: veritas
PostgreSQL Exporter :9187 service: postgres
Redis Exporter :9121 service: redis
Elasticsearch Exporter :9114 service: elasticsearch
Traefik :8082/metrics service: traefik

5.2 Key Metrics to Monitor#

API Service#

Metric Warning Threshold Critical Threshold Action
HTTP response time (p95) > 500ms > 2000ms Scale up or investigate DB
HTTP error rate (5xx) > 1% > 5% Check logs, rollback if needed
Rate limit rejections (429s) > 50/min > 200/min Review rate limit config
Active connections > 80% of max > 95% of max Scale up replicas

Database#

Metric Warning Threshold Critical Threshold Action
Connection pool utilization > 70% > 90% Increase pool size or scale
Query latency (p95) > 100ms > 500ms Analyze slow queries
Disk usage > 70% > 85% Expand storage or archive
Replication lag > 5s > 30s Check replica health
Dead tuples ratio > 10% > 20% Run VACUUM ANALYZE

Redis#

Metric Warning Threshold Critical Threshold Action
Memory usage > 70% > 85% Review eviction policies
Connected clients > 80% of max > 95% of max Scale or review connections
Queue depth (BullMQ) > 1000 jobs > 5000 jobs Scale workers
Failed jobs (BullMQ) > 10/hour > 50/hour Investigate failures

Ingestion Pipeline#

Metric Warning Threshold Critical Threshold Action
Feed consecutive failures > 5 > 10 Check feed URL, disable feed
Items processed per minute < 10 < 1 Check worker health
Unprocessed feed items backlog > 500 > 2000 Scale workers
Scheduler tick duration > 120s > 300s Reduce RSS concurrency

5.3 Grafana Dashboards#

Access Grafana at http://localhost:3100 (development) or the production Grafana URL.

Default credentials: admin/admin (change in production).

Dashboards are auto-provisioned from /var/lib/grafana/dashboards.

5.4 Distributed Tracing#

Jaeger provides distributed tracing:

  • UI: http://localhost:16686 (development)
  • Collector: port 14268
  • Agent: port 6831/UDP
  • Zipkin compatible: port 9411

Use traces to investigate:

  • Cross-service latency
  • Database query patterns
  • External API call chains
  • Error propagation paths

5.5 Log Analysis#

All services emit structured JSON logs. Key fields:

json
{
  "at": "2025-01-26T12:00:00.000Z",
  "event": "event_name",
  "level": "info|warn|error",
  "message": "Human-readable description",
  "error": "Error message (if applicable)",
  "stack": "Stack trace (development mode only)"
}

Ingestion Service Log Events#

Event Level Meaning
scheduler_tick_complete info Scheduler finished a collection cycle
producer_tick_complete info Producer finished enqueuing jobs
job_completed info Worker successfully processed an item
job_failed error Worker failed to process an item
worker_error error Worker encountered an unexpected error
feed_fetch_failed warn Individual feed fetch failed
scrape_failed warn Content scraping failed for an item

Useful Log Queries#

bash
# Find all errors in the last hour
journalctl -u veritas-api --since "1 hour ago" | jq 'select(.level == "error")'

# Find rate limit violations
journalctl -u veritas-api --since "1 hour ago" | jq 'select(.status == 429)'

# Find ingestion failures
journalctl -u veritas-ingestion --since "1 hour ago" | jq 'select(.event == "job_failed")'

# Count events by type
journalctl -u veritas-ingestion --since "1 hour ago" | jq -r '.event' | sort | uniq -c | sort -rn

6. Incident Response#

6.1 Severity Levels#

Severity Description Response Time Escalation
SEV-1 Complete service outage, data loss risk Immediate All hands on deck
SEV-2 Major functionality degraded, user-facing 15 minutes On-call + team lead
SEV-3 Minor functionality degraded, workaround exists 1 hour On-call engineer
SEV-4 Non-urgent issue, cosmetic, low impact Next business day Ticket in backlog

6.2 Incident Response Procedures#

SEV-1: Complete Service Outage#

  1. Acknowledge the incident in the alerting channel

  2. Assess scope:

    bash
    # Check all health endpoints
    curl https://veritas.oshun.io/health
    curl https://veritas.oshun.io/readyz
    curl https://veritas.oshun.io/health/detailed
    
    # Check Kubernetes pod status
    kubectl get pods -n oshun-production-veritas
    kubectl describe pod <pod-name> -n oshun-production-veritas
    
    # Check recent events
    kubectl get events -n oshun-production-veritas --sort-by='.lastTimestamp'
    
  3. Mitigate immediately:

    • If deployment-related: rollback to previous version
    • If infrastructure-related: check database, Redis, Elasticsearch status
    • If traffic-related: enable emergency rate limiting
  4. Communicate status to stakeholders

  5. Resolve the root cause

  6. Post-mortem within 48 hours

SEV-2: Degraded Performance#

  1. Identify the degraded component via /health/detailed
  2. Check recent deployments and changes
  3. Scale if resource-constrained:
    bash
    kubectl scale deployment veritas-api --replicas=5 -n oshun-production-veritas
    
  4. Investigate logs and metrics for the affected component
  5. Resolve or apply temporary mitigation

6.3 Common Incident Scenarios#

API Returns 503 (Service Unavailable)#

Symptoms: /readyz returns unhealthy, users see 503 errors.

Diagnosis:

bash
# Check which dependency is down
curl https://veritas.oshun.io/readyz
# Response shows which checks failed: database, redis, elasticsearch

# Check database
kubectl exec -it $(kubectl get pod -l app=postgres -n oshun-production-veritas -o name) \
  -- pg_isready -U oshun

# Check Redis
kubectl exec -it $(kubectl get pod -l app=redis -n oshun-production-veritas -o name) \
  -- redis-cli ping

# Check Elasticsearch
curl http://elasticsearch:9200/_cluster/health

Resolution:

  • Database down: Check pod status, disk space, connection limits
  • Redis down: Check memory, restart if OOM killed
  • Elasticsearch down: Check cluster health, disk watermarks

High Error Rate (5xx Responses)#

Symptoms: Error rate exceeds 5%, alerts firing.

Diagnosis:

bash
# Check API logs for error patterns
kubectl logs -l app=veritas-api -n oshun-production-veritas --tail=100 | \
  jq 'select(.level == "error")'

# Check for OOM kills
kubectl get events -n oshun-production-veritas | grep OOMKilled

# Check resource usage
kubectl top pods -n oshun-production-veritas

Resolution:

  • OOM kills: Increase memory limits
  • Database errors: Check connection pool exhaustion
  • Unhandled exceptions: Deploy hotfix or rollback

Ingestion Pipeline Stalled#

Symptoms: No new articles appearing, unprocessed backlog growing.

Diagnosis:

bash
# Check unprocessed items count
psql $VERITAS_DATABASE_URL -c "
  SELECT COUNT(*) as unprocessed
  FROM feed_items fi
  WHERE NOT EXISTS (
    SELECT 1 FROM normalized_feed_items nfi
    WHERE nfi.feed_item_id = fi.id
  );
"

# Check BullMQ queue depth
redis-cli LLEN bull:veritas:content-processing:wait
redis-cli LLEN bull:veritas:content-processing:active
redis-cli ZCARD bull:veritas:content-processing:failed

# Check scheduler logs
kubectl logs -l app=veritas-scheduler -n oshun-production-veritas --tail=50

# Check worker logs
kubectl logs -l app=veritas-worker -n oshun-production-veritas --tail=50

Resolution:

  • Scheduler stopped: Restart scheduler process
  • Workers stopped: Restart worker processes
  • Queue full of failed jobs: Investigate failure reasons, retry or clear
  • All feeds failing: Check network connectivity, DNS resolution

Rate Limiting Too Aggressive#

Symptoms: Legitimate users receiving 429 responses.

Diagnosis:

bash
# Check current rate limit configuration
echo "RATE_LIMIT_MAX=$RATE_LIMIT_MAX RATE_LIMIT_WINDOW_SECONDS=$RATE_LIMIT_WINDOW_SECONDS"

# Check Redis for rate limit keys
redis-cli KEYS "rl:*" | head -20
redis-cli ZCARD "rl:<client-ip>"

Resolution:

  • Increase RATE_LIMIT_MAX or RATE_LIMIT_WINDOW_SECONDS
  • Whitelist specific API keys from rate limiting
  • Scale API replicas to distribute load

7. Database Operations#

7.1 Connection Management#

PostgreSQL is configured with multiple databases for domain isolation:

Database Purpose
oshun_dev Main development database
veritas Veritas production data

Extensions enabled on all databases:

  • uuid-ossp — UUID generation
  • pgcrypto — Cryptographic functions
  • vector — pgvector for embeddings
  • pg_trgm — Trigram text similarity

Schemas per database: public, app, analytics, staging

7.2 Database Connection Strings#

bash
# Development
postgresql://oshun:oshun_dev@localhost:5432/oshun_dev

# Veritas-specific
postgresql://oshun:oshun_dev@localhost:5432/veritas

7.3 Backup Procedures#

RDS Automated Backups (AWS)#

Environment Retention Multi-AZ Deletion Protection
Production 30 days Yes Yes
Staging 7 days No No

Manual Backup#

bash
# Full database dump
pg_dump $VERITAS_DATABASE_URL -Fc -f veritas_backup_$(date +%Y%m%d_%H%M%S).dump

# Schema-only backup
pg_dump $VERITAS_DATABASE_URL --schema-only -f veritas_schema_$(date +%Y%m%d).sql

# Data-only backup for specific table
pg_dump $VERITAS_DATABASE_URL -t feed_items --data-only -f feed_items_backup.sql

Restore from Backup#

bash
# Restore full database
pg_restore -d $VERITAS_DATABASE_URL -c veritas_backup_20250126_120000.dump

# Restore specific table
pg_restore -d $VERITAS_DATABASE_URL -t feed_items veritas_backup.dump

7.4 Database Maintenance#

Routine Maintenance Queries#

sql
-- Check table sizes
SELECT schemaname, tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as total_size,
  pg_size_pretty(pg_relation_size(schemaname||'.'||tablename)) as table_size,
  pg_size_pretty(pg_indexes_size(schemaname||'.'||tablename::regclass)) as index_size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 20;

-- Check dead tuples (need VACUUM if high)
SELECT schemaname, relname, n_dead_tup, n_live_tup,
  round(n_dead_tup::numeric / NULLIF(n_live_tup, 0) * 100, 2) as dead_pct,
  last_vacuum, last_autovacuum
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC
LIMIT 20;

-- Check active connections
SELECT datname, count(*) as connections,
  count(*) FILTER (WHERE state = 'active') as active,
  count(*) FILTER (WHERE state = 'idle') as idle,
  count(*) FILTER (WHERE state = 'idle in transaction') as idle_in_txn
FROM pg_stat_activity
GROUP BY datname;

-- Check long-running queries
SELECT pid, now() - pg_stat_activity.query_start AS duration,
  query, state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes'
  AND state != 'idle'
ORDER BY duration DESC;

-- Check index usage
SELECT schemaname, tablename, indexname,
  idx_scan, idx_tup_read, idx_tup_fetch,
  pg_size_pretty(pg_relation_size(indexrelid)) as index_size
FROM pg_stat_user_indexes
ORDER BY idx_scan ASC
LIMIT 20;

Periodic Maintenance Tasks#

bash
# Analyze all tables (update query planner statistics)
psql $VERITAS_DATABASE_URL -c "ANALYZE;"

# Vacuum analyze specific table
psql $VERITAS_DATABASE_URL -c "VACUUM ANALYZE feed_items;"

# Reindex bloated indexes
psql $VERITAS_DATABASE_URL -c "REINDEX TABLE CONCURRENTLY feed_items;"

7.5 Database Schema Initialization#

The PostgreSQL initialization script (scripts/init-multiple-databases.sh) automatically:

  1. Creates databases from POSTGRES_MULTIPLE_DATABASES env var
  2. Enables extensions: uuid-ossp, pgcrypto, vector, pg_trgm
  3. Creates schemas: app, analytics, staging
  4. Sets schema permissions and default privileges
  5. Installs vector search helper functions:
    • vector_cosine_similarity(vector, vector) → float
    • vector_l2_distance(vector, vector) → float
    • suggest_vector_index() → index creation SQL

8. Queue and Worker Operations#

8.1 BullMQ Queue Architecture#

Queue Purpose Consumer
veritas:content-processing Feed item processing jobs Ingestion workers

8.2 Job Configuration#

Setting Value Description
Job type process_feed_item Processes a single feed item
Max attempts 3 Total attempts before failure
Backoff strategy Exponential 5000ms base delay
Completed job retention 24 hours Auto-removed after 24h
Completed job limit 10,000 Max completed jobs in Redis
Failed job retention 7 days Failed jobs kept for analysis
Job ID format process:{uuid} Prevents duplicate enqueuing

8.3 Queue Monitoring Commands#

bash
# Check queue sizes
redis-cli LLEN bull:veritas:content-processing:wait      # Waiting jobs
redis-cli LLEN bull:veritas:content-processing:active     # In-progress jobs
redis-cli ZCARD bull:veritas:content-processing:delayed   # Delayed/retry jobs
redis-cli ZCARD bull:veritas:content-processing:failed    # Failed jobs
redis-cli ZCARD bull:veritas:content-processing:completed # Completed jobs

# Get failed job details
redis-cli ZRANGE bull:veritas:content-processing:failed 0 9 WITHSCORES

# Get specific job data
redis-cli HGETALL bull:veritas:content-processing:<job-id>

# Check worker activity
redis-cli SMEMBERS bull:veritas:content-processing:workers

8.4 Queue Recovery Operations#

Clear Failed Jobs#

bash
# Remove all failed jobs
redis-cli DEL bull:veritas:content-processing:failed

# Retry specific failed job
# (requires application-level retry logic or BullMQ dashboard)

Drain Queue#

bash
# Remove all waiting jobs (emergency only)
redis-cli DEL bull:veritas:content-processing:wait

# Remove all delayed jobs
redis-cli DEL bull:veritas:content-processing:delayed

Reset Queue Completely#

bash
# WARNING: This removes all queue state
redis-cli KEYS "bull:veritas:content-processing:*" | xargs redis-cli DEL

9. Ingestion Service Operations#

9.1 Process Architecture#

The ingestion service runs as three independent processes:

text
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Scheduler  │     │   Producer   │     │    Worker     │
│              │     │              │     │              │
│ Fetches RSS  │────→│ Enqueues     │────→│ Processes    │
│ and sitemaps │ DB  │ unprocessed  │Queue│ feed items   │
│              │     │ items        │     │              │
└──────────────┘     └──────────────┘     └──────────────┘
       │                    │                    │
       ▼                    ▼                    ▼
   PostgreSQL         Redis (BullMQ)        PostgreSQL

9.2 Starting and Stopping#

bash
# Start individual processes
node dist/apps/veritas/ingestion/main.js scheduler
node dist/apps/veritas/ingestion/main.js producer
node dist/apps/veritas/ingestion/main.js worker

# Run single tick (useful for debugging)
VERITAS_ONCE=true node dist/apps/veritas/ingestion/main.js scheduler
VERITAS_ONCE=true node dist/apps/veritas/ingestion/main.js producer

# Seed database with Ghana news sources
node dist/apps/veritas/ingestion/main.js seed

# Dry-run seed (preview without database changes)
node dist/apps/veritas/ingestion/main.js seed --dry-run

# Seed summary only (no database connection needed)
node dist/apps/veritas/ingestion/main.js seed --summary

All processes handle SIGTERM and SIGINT for graceful shutdown, closing:

  1. Active workers/intervals
  2. Redis connections
  3. Database connection pools

9.3 Processing Pipeline Stages#

Each feed item goes through six stages:

Stage Purpose Output Table
1. Normalization Extract clean text from HTML/scrape normalized_feed_items
2. Deduplication Simhash64 fingerprinting + LSH bucketing content_fingerprints, fingerprint_buckets, dedupe_mappings
3. Language Detection Detect language from first 120 chars language_detections
4. Timestamp Normalize Resolve publication time with timezone timestamp_normalizations
5. Media Extraction Extract images, videos, audio from HTML media_assets
6. Source Attribution Link content to original sources source_attributions

9.4 Feed Management#

Check Feed Health#

sql
-- Feeds with high failure rates
SELECT s.name as source_name, sf.url, sf.consecutive_failures,
  sf.last_success_at, sf.last_fetched_at, sf.status
FROM source_feeds sf
JOIN sources s ON s.id = sf.source_id
WHERE sf.consecutive_failures > 3
ORDER BY sf.consecutive_failures DESC;

-- Feed collection summary
SELECT sf.status, COUNT(*) as count,
  AVG(sf.consecutive_failures) as avg_failures,
  MAX(sf.last_fetched_at) as most_recent_fetch
FROM source_feeds sf
GROUP BY sf.status;

-- Feeds due for collection
SELECT COUNT(*) as due_feeds
FROM source_feeds
WHERE status = 'active' AND next_fetch_at <= NOW();

Disable/Enable Feeds#

sql
-- Disable a problematic feed
UPDATE source_feeds
SET status = 'inactive'
WHERE url = 'https://example.com/rss';

-- Re-enable a feed and reset failures
UPDATE source_feeds
SET status = 'active', consecutive_failures = 0,
    next_fetch_at = NOW()
WHERE url = 'https://example.com/rss';

-- Suspend all feeds from a source
UPDATE source_feeds
SET status = 'suspended'
WHERE source_id = (SELECT id FROM sources WHERE canonical_domain = 'example.com');

9.5 Deduplication System#

The ingestion pipeline uses Simhash64 with LSH (Locality-Sensitive Hashing) for content deduplication:

  • Fingerprinting: 64-bit Simhash of normalized text
  • LSH Buckets: 4 bands for approximate nearest-neighbor lookup
  • Match Criteria: Min length ratio 0.6, max Hamming distance 3 bits
  • Canonical Mapping: Duplicate items map to a canonical item ID

Check Deduplication Stats#

sql
-- Deduplication rate
SELECT
  COUNT(*) as total_items,
  COUNT(DISTINCT dm.canonical_normalized_feed_item_id) as unique_items,
  COUNT(*) - COUNT(DISTINCT dm.canonical_normalized_feed_item_id) as duplicates,
  ROUND((COUNT(*) - COUNT(DISTINCT dm.canonical_normalized_feed_item_id))::numeric
    / COUNT(*) * 100, 2) as dedup_pct
FROM dedupe_mappings dm;

-- Items with most duplicates
SELECT dm.canonical_normalized_feed_item_id, COUNT(*) as duplicate_count,
  nfi.title
FROM dedupe_mappings dm
JOIN normalized_feed_items nfi ON nfi.id = dm.canonical_normalized_feed_item_id
GROUP BY dm.canonical_normalized_feed_item_id, nfi.title
ORDER BY duplicate_count DESC
LIMIT 20;

9.6 Backoff and Rate Limiting#

The scheduler implements adaptive backoff for failed feeds:

  • Exponential backoff: 2^failures * 60000ms (capped)
  • Retry-After respect: Honors HTTP 429/503 Retry-After headers
  • Conditional requests: Uses ETag and Last-Modified for 304 optimization
  • EWMA latency tracking: Adapts polling intervals based on response times

10. Troubleshooting Guide#

10.1 API Issues#

API Not Starting#

bash
# Check if port is in use
lsof -i :3001

# Check environment variables
env | grep -E "^(PORT|HOST|DATABASE_URL|REDIS_URL)"

# Test database connectivity directly
psql $DATABASE_URL -c "SELECT 1;"

# Test Redis connectivity
redis-cli -u $REDIS_URL ping

# Check Node.js version
node --version  # Should be 22+

Slow API Responses#

bash
# Check detailed health (includes latency info)
curl https://veritas.oshun.io/health/detailed

# Check database query performance
psql $DATABASE_URL -c "
  SELECT query, calls, mean_exec_time, total_exec_time
  FROM pg_stat_statements
  ORDER BY mean_exec_time DESC
  LIMIT 10;
"

# Check connection pool status
psql $DATABASE_URL -c "
  SELECT count(*), state
  FROM pg_stat_activity
  WHERE datname = 'veritas'
  GROUP BY state;
"

JWT Authentication Failures#

bash
# Verify token secret is set
echo $VERITAS_ACCESS_TOKEN_SECRET | wc -c  # Should be > 0

# Decode a JWT (header and payload only)
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq .

# Check token expiration
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq '.exp | todate'

10.2 Ingestion Issues#

Feeds Not Being Collected#

bash
# Check scheduler is running
ps aux | grep "veritas.*scheduler"

# Run a single tick manually for debugging
VERITAS_ONCE=true VERITAS_RSS_LIMIT=5 \
  node dist/apps/veritas/ingestion/main.js scheduler

# Check feed due dates
psql $VERITAS_DATABASE_URL -c "
  SELECT url, status, next_fetch_at, consecutive_failures
  FROM source_feeds
  WHERE status = 'active'
  ORDER BY next_fetch_at
  LIMIT 10;
"

Items Not Being Processed#

bash
# Check unprocessed count
psql $VERITAS_DATABASE_URL -c "
  SELECT COUNT(*) as unprocessed
  FROM feed_items fi
  WHERE NOT EXISTS (
    SELECT 1 FROM normalized_feed_items nfi
    WHERE nfi.feed_item_id = fi.id
  );
"

# Check queue status
redis-cli LLEN bull:veritas:content-processing:wait
redis-cli LLEN bull:veritas:content-processing:active

# Check worker is running
ps aux | grep "veritas.*worker"

# Run worker with verbose logging
LOG_LEVEL=debug node dist/apps/veritas/ingestion/main.js worker

Content Scraping Failures#

bash
# Test scraping a specific URL
VERITAS_ONCE=true VERITAS_SCRAPE_MISSING=true \
  node dist/apps/veritas/ingestion/main.js worker

# Check scrape results
psql $VERITAS_DATABASE_URL -c "
  SELECT fi.url, nfi.title, nfi.content_text IS NOT NULL as has_text,
    nfi.content_html IS NOT NULL as has_html
  FROM feed_items fi
  LEFT JOIN normalized_feed_items nfi ON nfi.feed_item_id = fi.id
  ORDER BY fi.created_at DESC
  LIMIT 20;
"

10.3 Infrastructure Issues#

PostgreSQL Connection Issues#

bash
# Check max connections
psql $DATABASE_URL -c "SHOW max_connections;"

# Check current connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# Kill long-running idle connections
psql $DATABASE_URL -c "
  SELECT pg_terminate_backend(pid)
  FROM pg_stat_activity
  WHERE state = 'idle'
  AND query_start < NOW() - interval '30 minutes'
  AND datname = 'veritas';
"

Redis Memory Issues#

bash
# Check Redis memory usage
redis-cli INFO memory | grep used_memory_human

# Check key count by pattern
redis-cli --scan --pattern "rl:*" | wc -l   # Rate limit keys
redis-cli --scan --pattern "bull:*" | wc -l  # BullMQ keys

# Clear rate limit data (if needed)
redis-cli --scan --pattern "rl:*" | xargs redis-cli DEL

# Check eviction policy
redis-cli CONFIG GET maxmemory-policy

Elasticsearch Issues#

bash
# Check cluster health
curl http://localhost:9200/_cluster/health?pretty

# Check index status
curl http://localhost:9200/_cat/indices?v

# Check disk watermarks
curl http://localhost:9200/_cluster/settings?include_defaults | \
  jq '.defaults.cluster.routing.allocation.disk'

11. Disaster Recovery#

11.1 Recovery Point Objective (RPO) and Recovery Time Objective (RTO)#

Component RPO RTO Backup Method
PostgreSQL 24 hours 1 hour RDS automated + manual
Redis Ephemeral 15 minutes Restart and repopulate
Elasticsearch Rebuild 2 hours Re-index from PostgreSQL
Feed items 0 (re-fetch) Variable Re-collect from sources
Configuration 0 (in Git) 15 minutes Redeploy from repository

11.2 Database Recovery#

bash
# Restore from RDS automated backup (AWS Console or CLI)
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier veritas-prod \
  --target-db-instance-identifier veritas-restored \
  --restore-time "2025-01-26T10:00:00Z"

# Restore from manual dump
pg_restore -d $NEW_DATABASE_URL -c veritas_backup.dump

11.3 Full Environment Recovery#

  1. Infrastructure: Apply Terraform configuration

    bash
    cd infrastructure/terraform
    terraform init
    terraform plan -var-file=environments/production.tfvars
    terraform apply -var-file=environments/production.tfvars
    
  2. Database: Restore from backup or RDS snapshot

  3. Application: Deploy from latest stable tag

    bash
    gh workflow run deploy.yml \
      -f environment=production \
      -f strategy=rolling
    
  4. Ingestion: Seed sources and restart collection

    bash
    node dist/apps/veritas/ingestion/main.js seed
    # Scheduler will automatically begin collecting from next_fetch_at
    
  5. Verification: Run full health checks

    bash
    curl https://veritas.oshun.io/readyz
    curl https://veritas.oshun.io/health/detailed
    

12. Security Operations#

12.1 Secret Rotation#

JWT Secret Rotation#

  1. Generate new secret:
    bash
    openssl rand -base64 64
    
  2. Update VERITAS_ACCESS_TOKEN_SECRET in environment configuration
  3. Deploy the change (rolling deployment)
  4. Existing tokens signed with the old secret will fail validation — users must re-authenticate

API Key Rotation#

  1. Generate new API key:
    bash
    openssl rand -hex 32
    
  2. Add new key to API_KEYS (comma-separated list supports multiple keys)
  3. Deploy the change
  4. Notify API consumers to switch to the new key
  5. Remove old key from API_KEYS after migration period

Tips Encryption Key Rotation#

The TIPS_ENCRYPTION_KEY is used for AES-256-GCM encryption of anonymous tips. Rotation requires re-encrypting existing tips:

  1. Generate new key:
    bash
    openssl rand -hex 16  # 32 hex chars = 16 bytes = 128 bits
    
  2. Write a migration script to decrypt with old key and re-encrypt with new key
  3. Update environment variable and deploy
  4. Verify decryption works with test tips

12.2 TLS Certificate Management#

Traefik handles TLS certificate management automatically via Let's Encrypt:

  • Challenge type: HTTP-01 (port 80)
  • Storage: /etc/traefik/acme/acme.json
  • Auto-renewal: Handled by Traefik

To force certificate renewal:

bash
# Remove existing certificates (Traefik will request new ones)
rm /etc/traefik/acme/acme.json
# Restart Traefik
docker restart traefik

12.3 Security Headers#

The API sets the following security headers on all responses:

  • X-Frame-Options: SAMEORIGIN
  • X-Content-Type-Options: nosniff
  • X-XSS-Protection: 1; mode=block
  • Referrer-Policy: strict-origin-when-cross-origin
  • Permissions-Policy: camera=(), microphone=(), geolocation=()

The Nginx web server adds equivalent headers for the frontend.

12.4 Rate Limiting Configuration#

The API uses sliding-window rate limiting with Redis sorted sets:

Setting Default Description
Window 60s Sliding window duration
Max requests 120 Maximum requests per window
Redis prefix rl: Key prefix for rate limit counters
Fallback In-memory Used when Redis is unavailable

Response headers:

  • X-RateLimit-Limit — Maximum requests allowed
  • X-RateLimit-Remaining — Requests remaining in current window
  • X-RateLimit-Reset — Unix timestamp when window resets

13. Runbook Checklists#

13.1 Pre-Deployment Checklist#

  • All CI checks pass (build, lint, test)
  • Trivy vulnerability scan shows no CRITICAL/HIGH issues
  • Database migrations (if any) are backward compatible
  • Environment variables are configured in target environment
  • Rollback plan documented and tested
  • Monitoring alerts are configured
  • Team notified of upcoming deployment

13.2 Post-Deployment Checklist#

  • Health endpoint returns 200 OK
  • Readiness endpoint returns ok for all dependencies
  • API response time < 2s at p95
  • No increase in error rate (5xx responses)
  • Rate limiting headers present in responses
  • Ingestion pipeline processing items normally
  • BullMQ queue depth is stable or decreasing
  • No new entries in error logs
  • Slack deployment notification received

13.3 Weekly Maintenance Checklist#

  • Review database table sizes and growth trends
  • Check dead tuple ratios and VACUUM status
  • Review feed health (consecutive failures)
  • Check Redis memory usage and eviction rate
  • Review Elasticsearch index sizes and shard health
  • Audit BullMQ failed job count and reasons
  • Review API error rate trends
  • Check TLS certificate expiration dates
  • Review and rotate stale API keys

13.4 Monthly Maintenance Checklist#

  • Review and update rate limit thresholds
  • Analyze slow query logs and optimize
  • Review database backup integrity (test restore)
  • Audit user access and API key usage
  • Review security scan results
  • Update dependencies (minor/patch versions)
  • Review monitoring alert thresholds
  • Capacity planning review (storage, compute, connections)

13.5 Incident Post-Mortem Template#

markdown
## Incident Report: [TITLE]

**Date**: YYYY-MM-DD **Duration**: HH:MM **Severity**: SEV-1/2/3/4 **Author**:
[Name]

### Summary

[One paragraph describing the incident]

### Timeline

- HH:MM — [Event]
- HH:MM — [Event]

### Root Cause

[Detailed technical explanation]

### Impact

- Users affected: [number/percentage]
- Duration of impact: [time]
- Data loss: [yes/no, details]

### Resolution

[Steps taken to resolve]

### Action Items

- [ ] [Action] — Owner — Due date
- [ ] [Action] — Owner — Due date

### Lessons Learned

[What went well, what could be improved]