Operational procedures for deploying, monitoring, and maintaining the Veritas AI News Agency platform.
Table of Contents#
- Service Overview
- Local Development Environment
- Deployment Procedures
- Service Operations
- Monitoring and Alerting
- Incident Response
- Database Operations
- Queue and Worker Operations
- Ingestion Service Operations
- Troubleshooting Guide
- Disaster Recovery
- Security Operations
- Runbook Checklists
1. Service Overview#
1.1 Service Inventory#
| Service | Type | Port | Health Endpoint | Description |
|---|---|---|---|---|
| veritas-api | HTTP API | 3001 | GET /health |
Hono REST API serving all endpoints |
| veritas-web | Web App | 3000 | GET /health |
Next.js 15 PWA frontend |
| veritas-ingestion | CLI Workers | N/A | N/A (process-based) | RSS/sitemap collection and pipeline |
| PostgreSQL | Database | 5432 | pg_isready |
Primary data store with pgvector |
| Redis | Cache/Queue | 6379 | redis-cli ping |
Rate limiting, BullMQ job queues |
| Elasticsearch | Search | 9200 | GET /_cluster/health |
Article full-text search |
1.2 Service Dependencies#
veritas-web ──→ veritas-api ──→ PostgreSQL
──→ Redis
──→ Elasticsearch
veritas-ingestion (scheduler) ──→ PostgreSQL
──→ External RSS feeds
veritas-ingestion (producer) ──→ PostgreSQL
──→ Redis (BullMQ)
veritas-ingestion (worker) ──→ PostgreSQL
──→ Redis (BullMQ)
1.3 Required Environment Variables#
API Service#
| Variable | Required | Default | Description |
|---|---|---|---|
PORT |
No | 3001 |
HTTP listen port |
HOST |
No | 0.0.0.0 |
HTTP listen host |
DATABASE_URL |
Yes | — | PostgreSQL connection string |
REDIS_URL |
No | — | Redis connection string |
ELASTICSEARCH_URL |
No | — | Elasticsearch connection string |
ELASTICSEARCH_ARTICLES_INDEX |
No | veritas-articles |
Elasticsearch index name |
VERITAS_ACCESS_TOKEN_SECRET |
Yes | — | JWT signing secret (HMAC-SHA256) |
NEXTAUTH_SECRET |
No | — | Fallback JWT secret |
TIPS_ENCRYPTION_KEY |
Yes | — | AES-256-GCM encryption key for tips |
LOG_LEVEL |
No | info |
Logging verbosity |
CORS_ENABLED |
No | false |
Enable CORS middleware |
CORS_ORIGINS |
No | * |
Comma-separated allowed origins |
RATE_LIMIT_ENABLED |
No | false |
Enable sliding-window rate limiting |
RATE_LIMIT_MAX |
No | 120 |
Max requests per window |
RATE_LIMIT_WINDOW_SECONDS |
No | 60 |
Rate limit window in seconds |
RATE_LIMIT_PREFIX |
No | rl: |
Redis key prefix for rate limiting |
API_KEY_AUTH_ENABLED |
No | false |
Enable API key authentication |
API_KEYS |
No | — | Comma-separated valid API keys |
API_KEY_HEADER |
No | x-api-key |
Header name for API key |
OPENAPI_ENABLED |
No | true |
Enable OpenAPI docs at /docs |
Ingestion Service#
| Variable | Required | Default | Description |
|---|---|---|---|
VERITAS_DATABASE_URL |
No | postgresql://oshun:oshun_dev@localhost:5432/veritas |
PostgreSQL connection string |
VERITAS_REDIS_URL |
No | redis://localhost:6379 |
Redis for BullMQ queues |
VERITAS_ONCE |
No | false |
Run single tick then exit |
VERITAS_POLL_INTERVAL_MS |
No | 60000 |
Scheduler/producer loop interval |
VERITAS_DISABLE_RSS |
No | false |
Disable RSS collection |
VERITAS_RSS_LIMIT |
No | 100 |
Max feeds per scheduler tick |
VERITAS_RSS_CONCURRENCY |
No | 10 |
Parallel feed fetches |
VERITAS_RSS_TIMEOUT_MS |
No | 30000 |
Per-feed fetch timeout |
VERITAS_RSS_MAX_BYTES |
No | 10485760 |
Max feed response size (10 MB) |
VERITAS_USER_AGENT |
No | VeritasBot/1.0 (+https://veritas.news) |
HTTP User-Agent string |
VERITAS_ENABLE_SITEMAPS |
No | false |
Enable XML sitemap collection |
VERITAS_SITEMAP_LIMIT |
No | 10 |
Max sitemaps per tick |
VERITAS_SITEMAP_CONCURRENCY |
No | 5 |
Parallel sitemap fetches |
VERITAS_SITEMAP_TIMEOUT_MS |
No | 60000 |
Per-sitemap fetch timeout |
VERITAS_SITEMAP_MAX_BYTES |
No | 52428800 |
Max sitemap response size (50 MB) |
VERITAS_SITEMAP_SINCE_DAYS |
No | 7 |
Only process entries from last N days |
VERITAS_SITEMAP_MAX_CHILDREN |
No | 50 |
Max child sitemaps to process |
VERITAS_ENABLE_SOCIAL |
No | false |
Enable social media collection |
VERITAS_SOCIAL_BEARER_TOKEN |
No | — | Social media API bearer token |
VERITAS_PRODUCER_LIMIT |
No | 100 |
Max items enqueued per tick |
VERITAS_WORKER_CONCURRENCY |
No | 4 |
Parallel job processing |
VERITAS_TIMEOUT_MS |
No | 30000 |
Per-job processing timeout |
VERITAS_MAX_BYTES |
No | 10485760 |
Max content size for scraping |
VERITAS_SCRAPE_MISSING |
No | true |
Scrape if no HTML in feed item |
2. Local Development Environment#
2.1 Prerequisites#
- Node.js 20+
- pnpm 10+
- Docker and Docker Compose
- Access to the Oshun monorepo
2.2 Starting Infrastructure#
# Start core infrastructure (PostgreSQL, Redis, MinIO, Mailpit)
docker compose -f docker/docker-compose.dev.yml up -d
# Verify services are healthy
docker compose -f docker/docker-compose.dev.yml ps
# Install dependencies
pnpm install
2.3 Starting Services#
# Start the API server (development mode with hot reload)
pnpm nx serve veritas-api
# Start the web frontend
pnpm nx serve veritas-web
# Start ingestion scheduler
pnpm nx run veritas-ingestion:scheduler
# Start ingestion producer
pnpm nx run veritas-ingestion:producer
# Start ingestion worker
pnpm nx run veritas-ingestion:worker
# Seed the database with Ghana news sources
pnpm nx run veritas-ingestion:seed
2.4 Optional Infrastructure Profiles#
# Elasticsearch for article search
docker compose -f docker/docker-compose.dev.yml --profile search up -d
# Qdrant for vector similarity
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d
# Kafka for streaming
docker compose -f docker/docker-compose.dev.yml --profile streaming up -d
# Prometheus + Grafana + Jaeger for observability
docker compose -f docker/docker-compose.dev.yml --profile observability up -d
# PgAdmin, Redis Commander, Kafka UI
docker compose -f docker/docker-compose.dev.yml --profile tools up -d
# Everything
docker compose -f docker/docker-compose.dev.yml --profile all up -d
2.5 Verifying Local Setup#
# Check API health
curl http://localhost:3001/health
# Check readiness (includes dependency checks)
curl http://localhost:3001/readyz
# Check detailed health with latencies
curl http://localhost:3001/health/detailed
# Verify database connectivity
docker exec oshun-dev-postgres pg_isready -U oshun
# Verify Redis
docker exec oshun-dev-redis redis-cli ping
3. Deployment Procedures#
3.1 Deployment Environments#
| Environment | Trigger | Strategy | Approval |
|---|---|---|---|
| Staging | Push to main |
Rolling deployment | Automatic |
| Production | Tag v*.*.* or release |
Canary (10%) → Blue-green → Full | Manual release |
| Hotfix | Manual workflow dispatch | Direct rolling or canary | Manual trigger |
3.2 CI/CD Pipeline Overview#
The deployment pipeline is defined in .github/workflows/deploy.yml (583 lines)
and consists of the following stages:
Push to main
│
├─→ Plan (determine affected apps, environment, strategy)
│
├─→ Build Images (matrix per affected app)
│ ├── Docker build with Buildx
│ ├── Multi-platform: linux/amd64, linux/arm64
│ ├── Trivy vulnerability scan (CRITICAL/HIGH)
│ └── Push to ghcr.io
│
├─→ Deploy Staging
│ ├── Helm upgrade --install
│ ├── Smoke tests (/health endpoints)
│ └── 10-minute timeout
│
├─→ Deploy Canary (production only, 10% traffic)
│ ├── Canary deployment with Ingress weights
│ ├── 5-minute soak validation
│ └── Multiple health checks
│
├─→ Deploy Production (full rollout)
│ ├── Blue-green: switch via service selector
│ │ OR
│ ├── Rolling: standard Helm upgrade
│ └── Post-deployment verification (health + perf < 2s)
│
└─→ Rollback (on failure)
├── kubectl rollout undo deployment
└── Slack notification
3.3 Docker Image Build#
All services use multi-stage Docker builds:
| Dockerfile | Base Image | Use Case | Health Check |
|---|---|---|---|
| Dockerfile.node | node:22-alpine | API and Node.js services | GET /health on port 3000 |
| Dockerfile.web | Nginx | SPA/PWA web applications | GET /health on port 8080 |
| Dockerfile.worker | node:22-alpine | Background job processors | SIGUSR1 signal |
| Dockerfile.gpu | CUDA 12.2 + Ubuntu | ML/AI workers | CUDA availability check |
Build characteristics:
- Non-root user (UID 1001:
oshun) dumb-initfor proper signal handling- pnpm frozen lockfile for deterministic builds
- Source maps enabled for debugging
- Prisma client generation when applicable
3.4 Staging Deployment#
# Automatic on push to main, or manual:
gh workflow run deploy.yml \
-f environment=staging \
-f strategy=rolling
# Verify staging deployment
curl https://staging.veritas.oshun.io/health
curl https://staging.veritas.oshun.io/readyz
3.5 Production Deployment#
# Option 1: Create a release tag
git tag v1.2.3
git push origin v1.2.3
# Option 2: Create a GitHub release
gh release create v1.2.3 --title "v1.2.3" --notes "Release notes"
# Option 3: Manual dispatch with canary
gh workflow run deploy.yml \
-f environment=production \
-f strategy=canary
3.6 Production Deployment Verification#
After a production deployment, verify:
# 1. Health check
curl https://veritas.oshun.io/health
# Expected: {"status":"ok","service":"veritas-api","timestamp":"..."}
# 2. Readiness check (dependency verification)
curl https://veritas.oshun.io/readyz
# Expected: {"status":"ok","checks":{"database":"ok","redis":"ok","elasticsearch":"ok"}}
# 3. Performance check (response time < 2s)
time curl -o /dev/null -s -w '%{time_total}' https://veritas.oshun.io/api/v1/articles?limit=5
# 4. API key authentication (if enabled)
curl -H "x-api-key: $API_KEY" https://veritas.oshun.io/api/v1/articles
# 5. Check rate limit headers
curl -I https://veritas.oshun.io/api/v1/articles
# Look for: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset
3.7 Rollback Procedures#
Automatic Rollback#
The CI/CD pipeline automatically rolls back on deployment failure:
deploy-production failure → rollback job triggers
→ kubectl rollout undo deployment
→ Slack notification sent
Manual Rollback#
# Rollback to previous Helm release
helm rollback veritas-api -n oshun-production-veritas
# Or rollback Kubernetes deployment directly
kubectl rollout undo deployment/veritas-api -n oshun-production-veritas
# Verify rollback
kubectl rollout status deployment/veritas-api -n oshun-production-veritas
curl https://veritas.oshun.io/health
Emergency Rollback (Docker)#
# List available image tags
docker images ghcr.io/greychimp/veritas-api --format "{{.Tag}}"
# Deploy specific previous version
helm upgrade veritas-api infrastructure/helm/oshun-service \
--set image.tag=v1.2.2 \
-n oshun-production-veritas \
-f infrastructure/helm/values/production.yaml
4. Service Operations#
4.1 API Service Startup Sequence#
The API service (apps/veritas/api/src/main.ts) initializes in this order:
- Read environment configuration
- Initialize PostgreSQL connection pool
- Initialize Redis client
- Initialize Elasticsearch client
- Create Hono application with middleware stack
- Start HTTP server on configured port
- Log startup information
Middleware stack (applied in order):
- Request logging
- Request timing (
X-Response-Timeheader) - Security headers
- CORS (if enabled)
- Rate limiting on
/api/*(if enabled, requires Redis) - API key authentication on
/api/*(if enabled) - Global error handler
4.2 API Service Graceful Shutdown#
The API handles SIGTERM and SIGINT signals:
- Close HTTP server (stop accepting new connections)
- Close PostgreSQL connection pool
- Close Redis connection (
quit()withdisconnect()fallback) - Close Elasticsearch client
- Exit with code 0 (success) or 1 (error during shutdown)
Unhandled promise rejections and uncaught exceptions trigger the shutdown sequence automatically.
4.3 Health Check Endpoints#
| Endpoint | Purpose | Response Codes | Use Case |
|---|---|---|---|
GET /health |
Basic liveness | 200 | Load balancer health check |
GET /healthz |
K8s liveness probe | 200 | Kubernetes liveness probe |
GET /readyz |
K8s readiness probe | 200, 503 | Kubernetes readiness probe |
GET /health/detailed |
Full diagnostics | 200 | Monitoring dashboards |
Readiness Probe Response Statuses#
| Status | Meaning | HTTP Code |
|---|---|---|
ok |
All dependencies healthy | 200 |
degraded |
Some dependencies unavailable | 200 |
unhealthy |
All dependencies unavailable | 503 |
The readiness probe checks: database connectivity, Redis connectivity, and Elasticsearch cluster health.
4.4 Scaling Configuration#
Horizontal Pod Autoscaler (Kubernetes)#
# API service autoscaling
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 70
- type: Resource
resource:
name: memory
targetAverageUtilization: 80
Connection Pool Sizing#
| Service | Pool Size | Idle Timeout | Connect Timeout |
|---|---|---|---|
| API (PostgreSQL) | 20 | 30s | 10s |
| Ingestion | 10 | 30s | 10s |
Redis Connection Settings#
maxRetriesPerRequest: null(unlimited retries for BullMQ reliability)enableReadyCheck: false(performance optimization)- Exponential backoff retry:
delay = min(attempts * 1000, 30000)
5. Monitoring and Alerting#
5.1 Metrics Collection#
Prometheus scrapes metrics at 15-second intervals in production (30s in development):
| Target | Endpoint | Labels |
|---|---|---|
| veritas-api | :3001/metrics |
domain: veritas |
| PostgreSQL | Exporter :9187 |
service: postgres |
| Redis | Exporter :9121 |
service: redis |
| Elasticsearch | Exporter :9114 |
service: elasticsearch |
| Traefik | :8082/metrics |
service: traefik |
5.2 Key Metrics to Monitor#
API Service#
| Metric | Warning Threshold | Critical Threshold | Action |
|---|---|---|---|
| HTTP response time (p95) | > 500ms | > 2000ms | Scale up or investigate DB |
| HTTP error rate (5xx) | > 1% | > 5% | Check logs, rollback if needed |
| Rate limit rejections (429s) | > 50/min | > 200/min | Review rate limit config |
| Active connections | > 80% of max | > 95% of max | Scale up replicas |
Database#
| Metric | Warning Threshold | Critical Threshold | Action |
|---|---|---|---|
| Connection pool utilization | > 70% | > 90% | Increase pool size or scale |
| Query latency (p95) | > 100ms | > 500ms | Analyze slow queries |
| Disk usage | > 70% | > 85% | Expand storage or archive |
| Replication lag | > 5s | > 30s | Check replica health |
| Dead tuples ratio | > 10% | > 20% | Run VACUUM ANALYZE |
Redis#
| Metric | Warning Threshold | Critical Threshold | Action |
|---|---|---|---|
| Memory usage | > 70% | > 85% | Review eviction policies |
| Connected clients | > 80% of max | > 95% of max | Scale or review connections |
| Queue depth (BullMQ) | > 1000 jobs | > 5000 jobs | Scale workers |
| Failed jobs (BullMQ) | > 10/hour | > 50/hour | Investigate failures |
Ingestion Pipeline#
| Metric | Warning Threshold | Critical Threshold | Action |
|---|---|---|---|
| Feed consecutive failures | > 5 | > 10 | Check feed URL, disable feed |
| Items processed per minute | < 10 | < 1 | Check worker health |
| Unprocessed feed items backlog | > 500 | > 2000 | Scale workers |
| Scheduler tick duration | > 120s | > 300s | Reduce RSS concurrency |
5.3 Grafana Dashboards#
Access Grafana at http://localhost:3100 (development) or the production
Grafana URL.
Default credentials: admin/admin (change in production).
Dashboards are auto-provisioned from /var/lib/grafana/dashboards.
5.4 Distributed Tracing#
Jaeger provides distributed tracing:
- UI:
http://localhost:16686(development) - Collector: port 14268
- Agent: port 6831/UDP
- Zipkin compatible: port 9411
Use traces to investigate:
- Cross-service latency
- Database query patterns
- External API call chains
- Error propagation paths
5.5 Log Analysis#
All services emit structured JSON logs. Key fields:
{
"at": "2025-01-26T12:00:00.000Z",
"event": "event_name",
"level": "info|warn|error",
"message": "Human-readable description",
"error": "Error message (if applicable)",
"stack": "Stack trace (development mode only)"
}
Ingestion Service Log Events#
| Event | Level | Meaning |
|---|---|---|
scheduler_tick_complete |
info | Scheduler finished a collection cycle |
producer_tick_complete |
info | Producer finished enqueuing jobs |
job_completed |
info | Worker successfully processed an item |
job_failed |
error | Worker failed to process an item |
worker_error |
error | Worker encountered an unexpected error |
feed_fetch_failed |
warn | Individual feed fetch failed |
scrape_failed |
warn | Content scraping failed for an item |
Useful Log Queries#
# Find all errors in the last hour
journalctl -u veritas-api --since "1 hour ago" | jq 'select(.level == "error")'
# Find rate limit violations
journalctl -u veritas-api --since "1 hour ago" | jq 'select(.status == 429)'
# Find ingestion failures
journalctl -u veritas-ingestion --since "1 hour ago" | jq 'select(.event == "job_failed")'
# Count events by type
journalctl -u veritas-ingestion --since "1 hour ago" | jq -r '.event' | sort | uniq -c | sort -rn
6. Incident Response#
6.1 Severity Levels#
| Severity | Description | Response Time | Escalation |
|---|---|---|---|
| SEV-1 | Complete service outage, data loss risk | Immediate | All hands on deck |
| SEV-2 | Major functionality degraded, user-facing | 15 minutes | On-call + team lead |
| SEV-3 | Minor functionality degraded, workaround exists | 1 hour | On-call engineer |
| SEV-4 | Non-urgent issue, cosmetic, low impact | Next business day | Ticket in backlog |
6.2 Incident Response Procedures#
SEV-1: Complete Service Outage#
-
Acknowledge the incident in the alerting channel
-
Assess scope:
bash# Check all health endpoints curl https://veritas.oshun.io/health curl https://veritas.oshun.io/readyz curl https://veritas.oshun.io/health/detailed # Check Kubernetes pod status kubectl get pods -n oshun-production-veritas kubectl describe pod <pod-name> -n oshun-production-veritas # Check recent events kubectl get events -n oshun-production-veritas --sort-by='.lastTimestamp' -
Mitigate immediately:
- If deployment-related: rollback to previous version
- If infrastructure-related: check database, Redis, Elasticsearch status
- If traffic-related: enable emergency rate limiting
-
Communicate status to stakeholders
-
Resolve the root cause
-
Post-mortem within 48 hours
SEV-2: Degraded Performance#
- Identify the degraded component via
/health/detailed - Check recent deployments and changes
- Scale if resource-constrained:bash
kubectl scale deployment veritas-api --replicas=5 -n oshun-production-veritas - Investigate logs and metrics for the affected component
- Resolve or apply temporary mitigation
6.3 Common Incident Scenarios#
API Returns 503 (Service Unavailable)#
Symptoms: /readyz returns unhealthy, users see 503 errors.
Diagnosis:
# Check which dependency is down
curl https://veritas.oshun.io/readyz
# Response shows which checks failed: database, redis, elasticsearch
# Check database
kubectl exec -it $(kubectl get pod -l app=postgres -n oshun-production-veritas -o name) \
-- pg_isready -U oshun
# Check Redis
kubectl exec -it $(kubectl get pod -l app=redis -n oshun-production-veritas -o name) \
-- redis-cli ping
# Check Elasticsearch
curl http://elasticsearch:9200/_cluster/health
Resolution:
- Database down: Check pod status, disk space, connection limits
- Redis down: Check memory, restart if OOM killed
- Elasticsearch down: Check cluster health, disk watermarks
High Error Rate (5xx Responses)#
Symptoms: Error rate exceeds 5%, alerts firing.
Diagnosis:
# Check API logs for error patterns
kubectl logs -l app=veritas-api -n oshun-production-veritas --tail=100 | \
jq 'select(.level == "error")'
# Check for OOM kills
kubectl get events -n oshun-production-veritas | grep OOMKilled
# Check resource usage
kubectl top pods -n oshun-production-veritas
Resolution:
- OOM kills: Increase memory limits
- Database errors: Check connection pool exhaustion
- Unhandled exceptions: Deploy hotfix or rollback
Ingestion Pipeline Stalled#
Symptoms: No new articles appearing, unprocessed backlog growing.
Diagnosis:
# Check unprocessed items count
psql $VERITAS_DATABASE_URL -c "
SELECT COUNT(*) as unprocessed
FROM feed_items fi
WHERE NOT EXISTS (
SELECT 1 FROM normalized_feed_items nfi
WHERE nfi.feed_item_id = fi.id
);
"
# Check BullMQ queue depth
redis-cli LLEN bull:veritas:content-processing:wait
redis-cli LLEN bull:veritas:content-processing:active
redis-cli ZCARD bull:veritas:content-processing:failed
# Check scheduler logs
kubectl logs -l app=veritas-scheduler -n oshun-production-veritas --tail=50
# Check worker logs
kubectl logs -l app=veritas-worker -n oshun-production-veritas --tail=50
Resolution:
- Scheduler stopped: Restart scheduler process
- Workers stopped: Restart worker processes
- Queue full of failed jobs: Investigate failure reasons, retry or clear
- All feeds failing: Check network connectivity, DNS resolution
Rate Limiting Too Aggressive#
Symptoms: Legitimate users receiving 429 responses.
Diagnosis:
# Check current rate limit configuration
echo "RATE_LIMIT_MAX=$RATE_LIMIT_MAX RATE_LIMIT_WINDOW_SECONDS=$RATE_LIMIT_WINDOW_SECONDS"
# Check Redis for rate limit keys
redis-cli KEYS "rl:*" | head -20
redis-cli ZCARD "rl:<client-ip>"
Resolution:
- Increase
RATE_LIMIT_MAXorRATE_LIMIT_WINDOW_SECONDS - Whitelist specific API keys from rate limiting
- Scale API replicas to distribute load
7. Database Operations#
7.1 Connection Management#
PostgreSQL is configured with multiple databases for domain isolation:
| Database | Purpose |
|---|---|
oshun_dev |
Main development database |
veritas |
Veritas production data |
Extensions enabled on all databases:
uuid-ossp— UUID generationpgcrypto— Cryptographic functionsvector— pgvector for embeddingspg_trgm— Trigram text similarity
Schemas per database: public, app, analytics, staging
7.2 Database Connection Strings#
# Development
postgresql://oshun:oshun_dev@localhost:5432/oshun_dev
# Veritas-specific
postgresql://oshun:oshun_dev@localhost:5432/veritas
7.3 Backup Procedures#
RDS Automated Backups (AWS)#
| Environment | Retention | Multi-AZ | Deletion Protection |
|---|---|---|---|
| Production | 30 days | Yes | Yes |
| Staging | 7 days | No | No |
Manual Backup#
# Full database dump
pg_dump $VERITAS_DATABASE_URL -Fc -f veritas_backup_$(date +%Y%m%d_%H%M%S).dump
# Schema-only backup
pg_dump $VERITAS_DATABASE_URL --schema-only -f veritas_schema_$(date +%Y%m%d).sql
# Data-only backup for specific table
pg_dump $VERITAS_DATABASE_URL -t feed_items --data-only -f feed_items_backup.sql
Restore from Backup#
# Restore full database
pg_restore -d $VERITAS_DATABASE_URL -c veritas_backup_20250126_120000.dump
# Restore specific table
pg_restore -d $VERITAS_DATABASE_URL -t feed_items veritas_backup.dump
7.4 Database Maintenance#
Routine Maintenance Queries#
-- Check table sizes
SELECT schemaname, tablename,
pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as total_size,
pg_size_pretty(pg_relation_size(schemaname||'.'||tablename)) as table_size,
pg_size_pretty(pg_indexes_size(schemaname||'.'||tablename::regclass)) as index_size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 20;
-- Check dead tuples (need VACUUM if high)
SELECT schemaname, relname, n_dead_tup, n_live_tup,
round(n_dead_tup::numeric / NULLIF(n_live_tup, 0) * 100, 2) as dead_pct,
last_vacuum, last_autovacuum
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC
LIMIT 20;
-- Check active connections
SELECT datname, count(*) as connections,
count(*) FILTER (WHERE state = 'active') as active,
count(*) FILTER (WHERE state = 'idle') as idle,
count(*) FILTER (WHERE state = 'idle in transaction') as idle_in_txn
FROM pg_stat_activity
GROUP BY datname;
-- Check long-running queries
SELECT pid, now() - pg_stat_activity.query_start AS duration,
query, state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes'
AND state != 'idle'
ORDER BY duration DESC;
-- Check index usage
SELECT schemaname, tablename, indexname,
idx_scan, idx_tup_read, idx_tup_fetch,
pg_size_pretty(pg_relation_size(indexrelid)) as index_size
FROM pg_stat_user_indexes
ORDER BY idx_scan ASC
LIMIT 20;
Periodic Maintenance Tasks#
# Analyze all tables (update query planner statistics)
psql $VERITAS_DATABASE_URL -c "ANALYZE;"
# Vacuum analyze specific table
psql $VERITAS_DATABASE_URL -c "VACUUM ANALYZE feed_items;"
# Reindex bloated indexes
psql $VERITAS_DATABASE_URL -c "REINDEX TABLE CONCURRENTLY feed_items;"
7.5 Database Schema Initialization#
The PostgreSQL initialization script (scripts/init-multiple-databases.sh)
automatically:
- Creates databases from
POSTGRES_MULTIPLE_DATABASESenv var - Enables extensions:
uuid-ossp,pgcrypto,vector,pg_trgm - Creates schemas:
app,analytics,staging - Sets schema permissions and default privileges
- Installs vector search helper functions:
vector_cosine_similarity(vector, vector)→ floatvector_l2_distance(vector, vector)→ floatsuggest_vector_index()→ index creation SQL
8. Queue and Worker Operations#
8.1 BullMQ Queue Architecture#
| Queue | Purpose | Consumer |
|---|---|---|
veritas:content-processing |
Feed item processing jobs | Ingestion workers |
8.2 Job Configuration#
| Setting | Value | Description |
|---|---|---|
| Job type | process_feed_item |
Processes a single feed item |
| Max attempts | 3 | Total attempts before failure |
| Backoff strategy | Exponential | 5000ms base delay |
| Completed job retention | 24 hours | Auto-removed after 24h |
| Completed job limit | 10,000 | Max completed jobs in Redis |
| Failed job retention | 7 days | Failed jobs kept for analysis |
| Job ID format | process:{uuid} |
Prevents duplicate enqueuing |
8.3 Queue Monitoring Commands#
# Check queue sizes
redis-cli LLEN bull:veritas:content-processing:wait # Waiting jobs
redis-cli LLEN bull:veritas:content-processing:active # In-progress jobs
redis-cli ZCARD bull:veritas:content-processing:delayed # Delayed/retry jobs
redis-cli ZCARD bull:veritas:content-processing:failed # Failed jobs
redis-cli ZCARD bull:veritas:content-processing:completed # Completed jobs
# Get failed job details
redis-cli ZRANGE bull:veritas:content-processing:failed 0 9 WITHSCORES
# Get specific job data
redis-cli HGETALL bull:veritas:content-processing:<job-id>
# Check worker activity
redis-cli SMEMBERS bull:veritas:content-processing:workers
8.4 Queue Recovery Operations#
Clear Failed Jobs#
# Remove all failed jobs
redis-cli DEL bull:veritas:content-processing:failed
# Retry specific failed job
# (requires application-level retry logic or BullMQ dashboard)
Drain Queue#
# Remove all waiting jobs (emergency only)
redis-cli DEL bull:veritas:content-processing:wait
# Remove all delayed jobs
redis-cli DEL bull:veritas:content-processing:delayed
Reset Queue Completely#
# WARNING: This removes all queue state
redis-cli KEYS "bull:veritas:content-processing:*" | xargs redis-cli DEL
9. Ingestion Service Operations#
9.1 Process Architecture#
The ingestion service runs as three independent processes:
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Scheduler │ │ Producer │ │ Worker │
│ │ │ │ │ │
│ Fetches RSS │────→│ Enqueues │────→│ Processes │
│ and sitemaps │ DB │ unprocessed │Queue│ feed items │
│ │ │ items │ │ │
└──────────────┘ └──────────────┘ └──────────────┘
│ │ │
▼ ▼ ▼
PostgreSQL Redis (BullMQ) PostgreSQL
9.2 Starting and Stopping#
# Start individual processes
node dist/apps/veritas/ingestion/main.js scheduler
node dist/apps/veritas/ingestion/main.js producer
node dist/apps/veritas/ingestion/main.js worker
# Run single tick (useful for debugging)
VERITAS_ONCE=true node dist/apps/veritas/ingestion/main.js scheduler
VERITAS_ONCE=true node dist/apps/veritas/ingestion/main.js producer
# Seed database with Ghana news sources
node dist/apps/veritas/ingestion/main.js seed
# Dry-run seed (preview without database changes)
node dist/apps/veritas/ingestion/main.js seed --dry-run
# Seed summary only (no database connection needed)
node dist/apps/veritas/ingestion/main.js seed --summary
All processes handle SIGTERM and SIGINT for graceful shutdown, closing:
- Active workers/intervals
- Redis connections
- Database connection pools
9.3 Processing Pipeline Stages#
Each feed item goes through six stages:
| Stage | Purpose | Output Table |
|---|---|---|
| 1. Normalization | Extract clean text from HTML/scrape | normalized_feed_items |
| 2. Deduplication | Simhash64 fingerprinting + LSH bucketing | content_fingerprints, fingerprint_buckets, dedupe_mappings |
| 3. Language Detection | Detect language from first 120 chars | language_detections |
| 4. Timestamp Normalize | Resolve publication time with timezone | timestamp_normalizations |
| 5. Media Extraction | Extract images, videos, audio from HTML | media_assets |
| 6. Source Attribution | Link content to original sources | source_attributions |
9.4 Feed Management#
Check Feed Health#
-- Feeds with high failure rates
SELECT s.name as source_name, sf.url, sf.consecutive_failures,
sf.last_success_at, sf.last_fetched_at, sf.status
FROM source_feeds sf
JOIN sources s ON s.id = sf.source_id
WHERE sf.consecutive_failures > 3
ORDER BY sf.consecutive_failures DESC;
-- Feed collection summary
SELECT sf.status, COUNT(*) as count,
AVG(sf.consecutive_failures) as avg_failures,
MAX(sf.last_fetched_at) as most_recent_fetch
FROM source_feeds sf
GROUP BY sf.status;
-- Feeds due for collection
SELECT COUNT(*) as due_feeds
FROM source_feeds
WHERE status = 'active' AND next_fetch_at <= NOW();
Disable/Enable Feeds#
-- Disable a problematic feed
UPDATE source_feeds
SET status = 'inactive'
WHERE url = 'https://example.com/rss';
-- Re-enable a feed and reset failures
UPDATE source_feeds
SET status = 'active', consecutive_failures = 0,
next_fetch_at = NOW()
WHERE url = 'https://example.com/rss';
-- Suspend all feeds from a source
UPDATE source_feeds
SET status = 'suspended'
WHERE source_id = (SELECT id FROM sources WHERE canonical_domain = 'example.com');
9.5 Deduplication System#
The ingestion pipeline uses Simhash64 with LSH (Locality-Sensitive Hashing) for content deduplication:
- Fingerprinting: 64-bit Simhash of normalized text
- LSH Buckets: 4 bands for approximate nearest-neighbor lookup
- Match Criteria: Min length ratio 0.6, max Hamming distance 3 bits
- Canonical Mapping: Duplicate items map to a canonical item ID
Check Deduplication Stats#
-- Deduplication rate
SELECT
COUNT(*) as total_items,
COUNT(DISTINCT dm.canonical_normalized_feed_item_id) as unique_items,
COUNT(*) - COUNT(DISTINCT dm.canonical_normalized_feed_item_id) as duplicates,
ROUND((COUNT(*) - COUNT(DISTINCT dm.canonical_normalized_feed_item_id))::numeric
/ COUNT(*) * 100, 2) as dedup_pct
FROM dedupe_mappings dm;
-- Items with most duplicates
SELECT dm.canonical_normalized_feed_item_id, COUNT(*) as duplicate_count,
nfi.title
FROM dedupe_mappings dm
JOIN normalized_feed_items nfi ON nfi.id = dm.canonical_normalized_feed_item_id
GROUP BY dm.canonical_normalized_feed_item_id, nfi.title
ORDER BY duplicate_count DESC
LIMIT 20;
9.6 Backoff and Rate Limiting#
The scheduler implements adaptive backoff for failed feeds:
- Exponential backoff:
2^failures * 60000ms(capped) - Retry-After respect: Honors HTTP 429/503
Retry-Afterheaders - Conditional requests: Uses
ETagandLast-Modifiedfor 304 optimization - EWMA latency tracking: Adapts polling intervals based on response times
10. Troubleshooting Guide#
10.1 API Issues#
API Not Starting#
# Check if port is in use
lsof -i :3001
# Check environment variables
env | grep -E "^(PORT|HOST|DATABASE_URL|REDIS_URL)"
# Test database connectivity directly
psql $DATABASE_URL -c "SELECT 1;"
# Test Redis connectivity
redis-cli -u $REDIS_URL ping
# Check Node.js version
node --version # Should be 22+
Slow API Responses#
# Check detailed health (includes latency info)
curl https://veritas.oshun.io/health/detailed
# Check database query performance
psql $DATABASE_URL -c "
SELECT query, calls, mean_exec_time, total_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
"
# Check connection pool status
psql $DATABASE_URL -c "
SELECT count(*), state
FROM pg_stat_activity
WHERE datname = 'veritas'
GROUP BY state;
"
JWT Authentication Failures#
# Verify token secret is set
echo $VERITAS_ACCESS_TOKEN_SECRET | wc -c # Should be > 0
# Decode a JWT (header and payload only)
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq .
# Check token expiration
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq '.exp | todate'
10.2 Ingestion Issues#
Feeds Not Being Collected#
# Check scheduler is running
ps aux | grep "veritas.*scheduler"
# Run a single tick manually for debugging
VERITAS_ONCE=true VERITAS_RSS_LIMIT=5 \
node dist/apps/veritas/ingestion/main.js scheduler
# Check feed due dates
psql $VERITAS_DATABASE_URL -c "
SELECT url, status, next_fetch_at, consecutive_failures
FROM source_feeds
WHERE status = 'active'
ORDER BY next_fetch_at
LIMIT 10;
"
Items Not Being Processed#
# Check unprocessed count
psql $VERITAS_DATABASE_URL -c "
SELECT COUNT(*) as unprocessed
FROM feed_items fi
WHERE NOT EXISTS (
SELECT 1 FROM normalized_feed_items nfi
WHERE nfi.feed_item_id = fi.id
);
"
# Check queue status
redis-cli LLEN bull:veritas:content-processing:wait
redis-cli LLEN bull:veritas:content-processing:active
# Check worker is running
ps aux | grep "veritas.*worker"
# Run worker with verbose logging
LOG_LEVEL=debug node dist/apps/veritas/ingestion/main.js worker
Content Scraping Failures#
# Test scraping a specific URL
VERITAS_ONCE=true VERITAS_SCRAPE_MISSING=true \
node dist/apps/veritas/ingestion/main.js worker
# Check scrape results
psql $VERITAS_DATABASE_URL -c "
SELECT fi.url, nfi.title, nfi.content_text IS NOT NULL as has_text,
nfi.content_html IS NOT NULL as has_html
FROM feed_items fi
LEFT JOIN normalized_feed_items nfi ON nfi.feed_item_id = fi.id
ORDER BY fi.created_at DESC
LIMIT 20;
"
10.3 Infrastructure Issues#
PostgreSQL Connection Issues#
# Check max connections
psql $DATABASE_URL -c "SHOW max_connections;"
# Check current connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"
# Kill long-running idle connections
psql $DATABASE_URL -c "
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
AND query_start < NOW() - interval '30 minutes'
AND datname = 'veritas';
"
Redis Memory Issues#
# Check Redis memory usage
redis-cli INFO memory | grep used_memory_human
# Check key count by pattern
redis-cli --scan --pattern "rl:*" | wc -l # Rate limit keys
redis-cli --scan --pattern "bull:*" | wc -l # BullMQ keys
# Clear rate limit data (if needed)
redis-cli --scan --pattern "rl:*" | xargs redis-cli DEL
# Check eviction policy
redis-cli CONFIG GET maxmemory-policy
Elasticsearch Issues#
# Check cluster health
curl http://localhost:9200/_cluster/health?pretty
# Check index status
curl http://localhost:9200/_cat/indices?v
# Check disk watermarks
curl http://localhost:9200/_cluster/settings?include_defaults | \
jq '.defaults.cluster.routing.allocation.disk'
11. Disaster Recovery#
11.1 Recovery Point Objective (RPO) and Recovery Time Objective (RTO)#
| Component | RPO | RTO | Backup Method |
|---|---|---|---|
| PostgreSQL | 24 hours | 1 hour | RDS automated + manual |
| Redis | Ephemeral | 15 minutes | Restart and repopulate |
| Elasticsearch | Rebuild | 2 hours | Re-index from PostgreSQL |
| Feed items | 0 (re-fetch) | Variable | Re-collect from sources |
| Configuration | 0 (in Git) | 15 minutes | Redeploy from repository |
11.2 Database Recovery#
# Restore from RDS automated backup (AWS Console or CLI)
aws rds restore-db-instance-to-point-in-time \
--source-db-instance-identifier veritas-prod \
--target-db-instance-identifier veritas-restored \
--restore-time "2025-01-26T10:00:00Z"
# Restore from manual dump
pg_restore -d $NEW_DATABASE_URL -c veritas_backup.dump
11.3 Full Environment Recovery#
-
Infrastructure: Apply Terraform configuration
bashcd infrastructure/terraform terraform init terraform plan -var-file=environments/production.tfvars terraform apply -var-file=environments/production.tfvars -
Database: Restore from backup or RDS snapshot
-
Application: Deploy from latest stable tag
bashgh workflow run deploy.yml \ -f environment=production \ -f strategy=rolling -
Ingestion: Seed sources and restart collection
bashnode dist/apps/veritas/ingestion/main.js seed # Scheduler will automatically begin collecting from next_fetch_at -
Verification: Run full health checks
bashcurl https://veritas.oshun.io/readyz curl https://veritas.oshun.io/health/detailed
12. Security Operations#
12.1 Secret Rotation#
JWT Secret Rotation#
- Generate new secret:bash
openssl rand -base64 64 - Update
VERITAS_ACCESS_TOKEN_SECRETin environment configuration - Deploy the change (rolling deployment)
- Existing tokens signed with the old secret will fail validation — users must re-authenticate
API Key Rotation#
- Generate new API key:bash
openssl rand -hex 32 - Add new key to
API_KEYS(comma-separated list supports multiple keys) - Deploy the change
- Notify API consumers to switch to the new key
- Remove old key from
API_KEYSafter migration period
Tips Encryption Key Rotation#
The TIPS_ENCRYPTION_KEY is used for AES-256-GCM encryption of anonymous tips.
Rotation requires re-encrypting existing tips:
- Generate new key:bash
openssl rand -hex 16 # 32 hex chars = 16 bytes = 128 bits - Write a migration script to decrypt with old key and re-encrypt with new key
- Update environment variable and deploy
- Verify decryption works with test tips
12.2 TLS Certificate Management#
Traefik handles TLS certificate management automatically via Let's Encrypt:
- Challenge type: HTTP-01 (port 80)
- Storage:
/etc/traefik/acme/acme.json - Auto-renewal: Handled by Traefik
To force certificate renewal:
# Remove existing certificates (Traefik will request new ones)
rm /etc/traefik/acme/acme.json
# Restart Traefik
docker restart traefik
12.3 Security Headers#
The API sets the following security headers on all responses:
X-Frame-Options: SAMEORIGINX-Content-Type-Options: nosniffX-XSS-Protection: 1; mode=blockReferrer-Policy: strict-origin-when-cross-originPermissions-Policy: camera=(), microphone=(), geolocation=()
The Nginx web server adds equivalent headers for the frontend.
12.4 Rate Limiting Configuration#
The API uses sliding-window rate limiting with Redis sorted sets:
| Setting | Default | Description |
|---|---|---|
| Window | 60s | Sliding window duration |
| Max requests | 120 | Maximum requests per window |
| Redis prefix | rl: |
Key prefix for rate limit counters |
| Fallback | In-memory | Used when Redis is unavailable |
Response headers:
X-RateLimit-Limit— Maximum requests allowedX-RateLimit-Remaining— Requests remaining in current windowX-RateLimit-Reset— Unix timestamp when window resets
13. Runbook Checklists#
13.1 Pre-Deployment Checklist#
- All CI checks pass (build, lint, test)
- Trivy vulnerability scan shows no CRITICAL/HIGH issues
- Database migrations (if any) are backward compatible
- Environment variables are configured in target environment
- Rollback plan documented and tested
- Monitoring alerts are configured
- Team notified of upcoming deployment
13.2 Post-Deployment Checklist#
- Health endpoint returns
200 OK - Readiness endpoint returns
okfor all dependencies - API response time < 2s at p95
- No increase in error rate (5xx responses)
- Rate limiting headers present in responses
- Ingestion pipeline processing items normally
- BullMQ queue depth is stable or decreasing
- No new entries in error logs
- Slack deployment notification received
13.3 Weekly Maintenance Checklist#
- Review database table sizes and growth trends
- Check dead tuple ratios and VACUUM status
- Review feed health (consecutive failures)
- Check Redis memory usage and eviction rate
- Review Elasticsearch index sizes and shard health
- Audit BullMQ failed job count and reasons
- Review API error rate trends
- Check TLS certificate expiration dates
- Review and rotate stale API keys
13.4 Monthly Maintenance Checklist#
- Review and update rate limit thresholds
- Analyze slow query logs and optimize
- Review database backup integrity (test restore)
- Audit user access and API key usage
- Review security scan results
- Update dependencies (minor/patch versions)
- Review monitoring alert thresholds
- Capacity planning review (storage, compute, connections)
13.5 Incident Post-Mortem Template#
## Incident Report: [TITLE]
**Date**: YYYY-MM-DD **Duration**: HH:MM **Severity**: SEV-1/2/3/4 **Author**:
[Name]
### Summary
[One paragraph describing the incident]
### Timeline
- HH:MM — [Event]
- HH:MM — [Event]
### Root Cause
[Detailed technical explanation]
### Impact
- Users affected: [number/percentage]
- Duration of impact: [time]
- Data loss: [yes/no, details]
### Resolution
[Steps taken to resolve]
### Action Items
- [ ] [Action] — Owner — Due date
- [ ] [Action] — Owner — Due date
### Lessons Learned
[What went well, what could be improved]