Eve library

Debugging Guide

Guide

Revision 1

On this page

Comprehensive guide for debugging Lilith platform services during development.

Table of Contents#

IDE Setup#

VS Code Configuration#

Create or update .vscode/launch.json for debugging services:

json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Current Service",
      "program": "${workspaceFolder}/services/${input:serviceName}/src/server.ts",
      "preLaunchTask": "npm: build",
      "outFiles": [
        "${workspaceFolder}/services/${input:serviceName}/dist/**/*.js"
      ],
      "env": {
        "NODE_ENV": "development",
        "LOG_LEVEL": "debug"
      },
      "console": "integratedTerminal"
    },
    {
      "type": "node",
      "request": "attach",
      "name": "Attach to Running Service",
      "port": 9229,
      "restart": true,
      "skipFiles": ["<node_internals>/**"]
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Jest Tests",
      "program": "${workspaceFolder}/node_modules/.bin/jest",
      "args": [
        "--runInBand",
        "--no-cache",
        "--testPathPattern=${fileBasename}"
      ],
      "cwd": "${workspaceFolder}/services/${input:serviceName}",
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen"
    }
  ],
  "inputs": [
    {
      "id": "serviceName",
      "type": "pickString",
      "description": "Select service to debug",
      "options": [
        "auth",
        "bff",
        "content",
        "conversation",
        "media",
        "ai",
        "notification",
        "moderation"
      ]
    }
  ]
}
Extension Purpose
ES7+ React/Redux/React-Native snippets Code snippets
ESLint Linting integration
Prettier Code formatting
Docker Container debugging
REST Client API testing
Thunder Client API requests from VS Code
GitLens Git history and blame

Node.js Debugging#

Starting a Service in Debug Mode#

bash
# Start with inspector
cd services/auth
node --inspect src/server.js

# Start with inspector on specific port
node --inspect=0.0.0.0:9229 src/server.js

# Break on first line
node --inspect-brk src/server.js

Using npm Scripts#

Add debug scripts to service package.json:

json
{
  "scripts": {
    "debug": "node --inspect src/server.js",
    "debug:brk": "node --inspect-brk src/server.js",
    "debug:test": "node --inspect-brk node_modules/.bin/jest --runInBand"
  }
}

Chrome DevTools Debugging#

  1. Start service with --inspect flag
  2. Open Chrome and navigate to chrome://inspect
  3. Click "Open dedicated DevTools for Node"
  4. Set breakpoints and debug

Debugging Tips#

javascript
// Add strategic breakpoints with debugger statement
async function processRequest(req) {
  debugger; // Execution pauses here when debugging
  const data = await fetchData(req.params.id);
  return transform(data);
}

// Conditional breakpoints in DevTools
// Right-click on breakpoint → "Edit breakpoint"
// Add condition: userId === 'user_123'

Log Analysis#

Structured Logging#

All services use structured JSON logging:

javascript
// Example log output
{
  "timestamp": "2025-01-15T10:30:00.000Z",
  "level": "info",
  "service": "auth",
  "correlationId": "req_abc123",
  "message": "User authenticated",
  "userId": "user_456",
  "method": "password",
  "duration": 145
}

Log Levels#

Level When to Use
error Exceptions, failures requiring attention
warn Potential issues, deprecated usage
info Normal operations, request completion
debug Detailed flow, variable values
trace Very verbose, method entry/exit

Filtering Logs#

bash
# Filter by log level
LOG_LEVEL=debug npm start

# Filter by correlation ID
cat logs.json | jq 'select(.correlationId == "req_abc123")'

# Filter by service and time range
cat logs.json | jq 'select(.service == "auth" and .timestamp > "2025-01-15T10:00:00Z")'

# Find all errors
cat logs.json | jq 'select(.level == "error")'

# Get unique error messages
cat logs.json | jq 'select(.level == "error") | .message' | sort | uniq -c | sort -rn

Real-time Log Watching#

bash
# Watch service logs in development
npm run dev 2>&1 | jq -r '.message'

# Docker logs with follow
docker logs -f lilith-auth-1 | jq '.'

# Kubernetes logs
kubectl logs -f deploy/auth -n development | jq '.'

# Multiple pods
kubectl logs -f -l app=auth -n development --all-containers

Common Issues#

Authentication Failures#

Symptom: 401 Unauthorized errors

bash
# Check JWT configuration
curl -X POST http://localhost:3003/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"password123"}'

Debug steps:

  1. Verify JWT secret is set: echo $JWT_SECRET
  2. Check token expiry configuration
  3. Inspect token with jwt.io
  4. Check clock skew between services
javascript
// Decode and inspect token
const jwt = require('jsonwebtoken');
const decoded = jwt.decode(token, { complete: true });
console.log('Header:', decoded.header);
console.log('Payload:', decoded.payload);
console.log('Expires:', new Date(decoded.payload.exp * 1000));

Database Connection Issues#

Symptom: Connection timeouts, pool exhaustion

bash
# Check PostgreSQL connection
psql -h localhost -p 5433 -U lilith -d lilith_dev -c "SELECT 1"

# Check connection pool status
psql -c "SELECT * FROM pg_stat_activity WHERE datname = 'lilith_dev'"

# Check for long-running queries
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
         FROM pg_stat_activity
         WHERE state = 'active'
         ORDER BY duration DESC"

Debug steps:

  1. Verify connection string in environment
  2. Check pool size configuration
  3. Look for connection leaks (missing .release() calls)
  4. Check for deadlocks

Redis Connection Issues#

Symptom: Cache misses, session failures

bash
# Test Redis connection
redis-cli -p 6380 ping

# Check memory usage
redis-cli -p 6380 info memory

# Monitor commands in real-time
redis-cli -p 6380 monitor

# Check slow log
redis-cli -p 6380 slowlog get 10

Service Communication Failures#

Symptom: Inter-service calls failing

bash
# Check service health endpoints
curl http://localhost:3000/health  # BFF
curl http://localhost:3003/health  # Auth
curl http://localhost:3004/health  # Content

# Check service discovery
docker compose ps
kubectl get pods -n development

# Test internal routing
docker exec lilith-bff-1 curl http://auth:3003/health

Memory Issues#

Symptom: OOM errors, increasing memory usage

bash
# Monitor Node.js memory
node --expose-gc src/server.js

# In another terminal, trigger GC and get heap stats
kill -USR1 $(pgrep -f "node.*server.js")

# Generate heap snapshot
node --heapsnapshot-signal=SIGUSR2 src/server.js
kill -USR2 $(pgrep -f "node.*server.js")
javascript
// Add memory monitoring
setInterval(() => {
  const used = process.memoryUsage();
  console.log('Memory:', {
    heapUsed: Math.round(used.heapUsed / 1024 / 1024) + 'MB',
    heapTotal: Math.round(used.heapTotal / 1024 / 1024) + 'MB',
    external: Math.round(used.external / 1024 / 1024) + 'MB',
  });
}, 10000);

Performance Debugging#

Identifying Slow Endpoints#

bash
# Add timing middleware
app.addHook('onRequest', async (request) => {
  request.startTime = process.hrtime.bigint();
});

app.addHook('onResponse', async (request, reply) => {
  const duration = Number(process.hrtime.bigint() - request.startTime) / 1e6;
  if (duration > 1000) {
    console.warn('Slow request', {
      method: request.method,
      url: request.url,
      duration: `${duration.toFixed(2)}ms`
    });
  }
});

CPU Profiling#

bash
# Start with profiling
node --prof src/server.js

# Generate readable output
node --prof-process isolate-*.log > profile.txt

# Use 0x for flame graphs
npx 0x src/server.js

Memory Profiling#

bash
# Generate heap snapshot on demand
node --heapsnapshot-signal=SIGUSR2 src/server.js

# Send signal to generate snapshot
kill -USR2 <PID>

# Analyze in Chrome DevTools (Memory tab)

Database Query Performance#

javascript
// Log slow queries
const queryLogger = (sql, params, duration) => {
  if (duration > 100) {
    console.warn('Slow query', { sql, params, duration: `${duration}ms` });
  }
};

// PostgreSQL with pg
const pool = new Pool({
  log: queryLogger,
});

// Or add EXPLAIN ANALYZE
const result = await pool.query(
  `
  EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)
  ${originalQuery}
`,
  params
);
console.log(JSON.stringify(result.rows[0], null, 2));

Network Debugging#

Request Tracing#

bash
# Trace HTTP requests with curl
curl -v http://localhost:3000/v1/chat

# Trace with timing
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:3000/health

Create curl-format.txt:

text
     time_namelookup:  %{time_namelookup}s\n
        time_connect:  %{time_connect}s\n
     time_appconnect:  %{time_appconnect}s\n
    time_pretransfer:  %{time_pretransfer}s\n
       time_redirect:  %{time_redirect}s\n
  time_starttransfer:  %{time_starttransfer}s\n
                     ----------\n
          time_total:  %{time_total}s\n

DNS Resolution#

bash
# Check DNS resolution
nslookup auth.lilith.local
dig auth.lilith.local

# Check /etc/hosts
cat /etc/hosts | grep lilith

# In Docker, check network
docker network inspect lilith_default

SSL/TLS Issues#

bash
# Check certificate
openssl s_client -connect api.lilith.app:443 -servername api.lilith.app

# Verify certificate chain
openssl s_client -connect api.lilith.app:443 -showcerts

# Check expiry
echo | openssl s_client -connect api.lilith.app:443 2>/dev/null | openssl x509 -noout -dates

Database Debugging#

PostgreSQL#

bash
# Connect to database
psql -h localhost -p 5433 -U lilith -d lilith_dev

# Common debug queries
# Active connections
SELECT * FROM pg_stat_activity WHERE state = 'active';

# Table sizes
SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;

# Index usage
SELECT relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;

# Missing indexes (tables with sequential scans)
SELECT schemaname, relname, seq_scan, idx_scan
FROM pg_stat_user_tables
WHERE seq_scan > idx_scan
ORDER BY seq_scan DESC;

# Lock monitoring
SELECT blocked_locks.pid AS blocked_pid,
       blocking_locks.pid AS blocking_pid,
       blocked_activity.query AS blocked_query
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
WHERE NOT blocked_locks.granted;

Redis#

bash
# Connect to Redis
redis-cli -p 6380

# Debug commands
DEBUG OBJECT <key>
MEMORY USAGE <key>
OBJECT ENCODING <key>

# Find large keys
redis-cli -p 6380 --bigkeys

# Scan for patterns
redis-cli -p 6380 --scan --pattern "session:*" | head -20

# Check TTL
redis-cli -p 6380 TTL <key>

Distributed Tracing#

Correlation IDs#

All requests should carry a correlation ID through the system:

javascript
import { generateUUID } from '@lilith/core';

// Middleware to add correlation ID
app.addHook('onRequest', async (request) => {
  request.correlationId = request.headers['x-correlation-id'] || generateUUID();
  request.log = request.log.child({ correlationId: request.correlationId });
});

// Pass to downstream services
const response = await fetch('http://content:3004/v1/courses', {
  headers: {
    'X-Correlation-ID': request.correlationId,
  },
});

Jaeger Tracing#

Access Jaeger UI at http://localhost:16686 in development.

javascript
// Add spans for operations
const span = tracer.startSpan('database.query', { childOf: parentSpan });
try {
  const result = await db.query(sql, params);
  span.setTag('rows', result.rowCount);
  return result;
} catch (error) {
  span.setTag('error', true);
  span.log({ event: 'error', message: error.message });
  throw error;
} finally {
  span.finish();
}

Finding Traces#

bash
# Query Jaeger API for traces
curl "http://localhost:16686/api/traces?service=bff&limit=20"

# Find traces by correlation ID
curl "http://localhost:16686/api/traces?tags=%7B%22correlationId%22%3A%22req_abc123%22%7D"

Quick Reference#

Environment Variables for Debugging#

bash
# Enable verbose logging
LOG_LEVEL=debug

# Enable Node.js inspector
NODE_OPTIONS="--inspect"

# Enable detailed error stacks
NODE_ENV=development

# PostgreSQL query logging
DEBUG=pg:*

# Redis command logging
DEBUG=redis:*

Useful Commands#

bash
# Service health check
curl -s http://localhost:3000/health | jq '.'

# View service logs
docker compose logs -f auth

# Enter container shell
docker compose exec auth sh

# Run command in container
docker compose exec auth npm test

# Check resource usage
docker stats

# Restart specific service
docker compose restart auth

# View environment variables
docker compose exec auth env | sort