Comprehensive guide for debugging Lilith platform services during development.
Table of Contents#
- IDE Setup
- Node.js Debugging
- Log Analysis
- Common Issues
- Performance Debugging
- Network Debugging
- Database Debugging
- Distributed Tracing
IDE Setup#
VS Code Configuration#
Create or update .vscode/launch.json for debugging services:
json
{
"version": "0.2.0",
"configurations": [
{
"type": "node",
"request": "launch",
"name": "Debug Current Service",
"program": "${workspaceFolder}/services/${input:serviceName}/src/server.ts",
"preLaunchTask": "npm: build",
"outFiles": [
"${workspaceFolder}/services/${input:serviceName}/dist/**/*.js"
],
"env": {
"NODE_ENV": "development",
"LOG_LEVEL": "debug"
},
"console": "integratedTerminal"
},
{
"type": "node",
"request": "attach",
"name": "Attach to Running Service",
"port": 9229,
"restart": true,
"skipFiles": ["<node_internals>/**"]
},
{
"type": "node",
"request": "launch",
"name": "Debug Jest Tests",
"program": "${workspaceFolder}/node_modules/.bin/jest",
"args": [
"--runInBand",
"--no-cache",
"--testPathPattern=${fileBasename}"
],
"cwd": "${workspaceFolder}/services/${input:serviceName}",
"console": "integratedTerminal",
"internalConsoleOptions": "neverOpen"
}
],
"inputs": [
{
"id": "serviceName",
"type": "pickString",
"description": "Select service to debug",
"options": [
"auth",
"bff",
"content",
"conversation",
"media",
"ai",
"notification",
"moderation"
]
}
]
}
Recommended VS Code Extensions#
| Extension | Purpose |
|---|---|
| ES7+ React/Redux/React-Native snippets | Code snippets |
| ESLint | Linting integration |
| Prettier | Code formatting |
| Docker | Container debugging |
| REST Client | API testing |
| Thunder Client | API requests from VS Code |
| GitLens | Git history and blame |
Node.js Debugging#
Starting a Service in Debug Mode#
bash
# Start with inspector
cd services/auth
node --inspect src/server.js
# Start with inspector on specific port
node --inspect=0.0.0.0:9229 src/server.js
# Break on first line
node --inspect-brk src/server.js
Using npm Scripts#
Add debug scripts to service package.json:
json
{
"scripts": {
"debug": "node --inspect src/server.js",
"debug:brk": "node --inspect-brk src/server.js",
"debug:test": "node --inspect-brk node_modules/.bin/jest --runInBand"
}
}
Chrome DevTools Debugging#
- Start service with
--inspectflag - Open Chrome and navigate to
chrome://inspect - Click "Open dedicated DevTools for Node"
- Set breakpoints and debug
Debugging Tips#
javascript
// Add strategic breakpoints with debugger statement
async function processRequest(req) {
debugger; // Execution pauses here when debugging
const data = await fetchData(req.params.id);
return transform(data);
}
// Conditional breakpoints in DevTools
// Right-click on breakpoint → "Edit breakpoint"
// Add condition: userId === 'user_123'
Log Analysis#
Structured Logging#
All services use structured JSON logging:
javascript
// Example log output
{
"timestamp": "2025-01-15T10:30:00.000Z",
"level": "info",
"service": "auth",
"correlationId": "req_abc123",
"message": "User authenticated",
"userId": "user_456",
"method": "password",
"duration": 145
}
Log Levels#
| Level | When to Use |
|---|---|
error |
Exceptions, failures requiring attention |
warn |
Potential issues, deprecated usage |
info |
Normal operations, request completion |
debug |
Detailed flow, variable values |
trace |
Very verbose, method entry/exit |
Filtering Logs#
bash
# Filter by log level
LOG_LEVEL=debug npm start
# Filter by correlation ID
cat logs.json | jq 'select(.correlationId == "req_abc123")'
# Filter by service and time range
cat logs.json | jq 'select(.service == "auth" and .timestamp > "2025-01-15T10:00:00Z")'
# Find all errors
cat logs.json | jq 'select(.level == "error")'
# Get unique error messages
cat logs.json | jq 'select(.level == "error") | .message' | sort | uniq -c | sort -rn
Real-time Log Watching#
bash
# Watch service logs in development
npm run dev 2>&1 | jq -r '.message'
# Docker logs with follow
docker logs -f lilith-auth-1 | jq '.'
# Kubernetes logs
kubectl logs -f deploy/auth -n development | jq '.'
# Multiple pods
kubectl logs -f -l app=auth -n development --all-containers
Common Issues#
Authentication Failures#
Symptom: 401 Unauthorized errors
bash
# Check JWT configuration
curl -X POST http://localhost:3003/v1/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"test@example.com","password":"password123"}'
Debug steps:
- Verify JWT secret is set:
echo $JWT_SECRET - Check token expiry configuration
- Inspect token with jwt.io
- Check clock skew between services
javascript
// Decode and inspect token
const jwt = require('jsonwebtoken');
const decoded = jwt.decode(token, { complete: true });
console.log('Header:', decoded.header);
console.log('Payload:', decoded.payload);
console.log('Expires:', new Date(decoded.payload.exp * 1000));
Database Connection Issues#
Symptom: Connection timeouts, pool exhaustion
bash
# Check PostgreSQL connection
psql -h localhost -p 5433 -U lilith -d lilith_dev -c "SELECT 1"
# Check connection pool status
psql -c "SELECT * FROM pg_stat_activity WHERE datname = 'lilith_dev'"
# Check for long-running queries
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC"
Debug steps:
- Verify connection string in environment
- Check pool size configuration
- Look for connection leaks (missing
.release()calls) - Check for deadlocks
Redis Connection Issues#
Symptom: Cache misses, session failures
bash
# Test Redis connection
redis-cli -p 6380 ping
# Check memory usage
redis-cli -p 6380 info memory
# Monitor commands in real-time
redis-cli -p 6380 monitor
# Check slow log
redis-cli -p 6380 slowlog get 10
Service Communication Failures#
Symptom: Inter-service calls failing
bash
# Check service health endpoints
curl http://localhost:3000/health # BFF
curl http://localhost:3003/health # Auth
curl http://localhost:3004/health # Content
# Check service discovery
docker compose ps
kubectl get pods -n development
# Test internal routing
docker exec lilith-bff-1 curl http://auth:3003/health
Memory Issues#
Symptom: OOM errors, increasing memory usage
bash
# Monitor Node.js memory
node --expose-gc src/server.js
# In another terminal, trigger GC and get heap stats
kill -USR1 $(pgrep -f "node.*server.js")
# Generate heap snapshot
node --heapsnapshot-signal=SIGUSR2 src/server.js
kill -USR2 $(pgrep -f "node.*server.js")
javascript
// Add memory monitoring
setInterval(() => {
const used = process.memoryUsage();
console.log('Memory:', {
heapUsed: Math.round(used.heapUsed / 1024 / 1024) + 'MB',
heapTotal: Math.round(used.heapTotal / 1024 / 1024) + 'MB',
external: Math.round(used.external / 1024 / 1024) + 'MB',
});
}, 10000);
Performance Debugging#
Identifying Slow Endpoints#
bash
# Add timing middleware
app.addHook('onRequest', async (request) => {
request.startTime = process.hrtime.bigint();
});
app.addHook('onResponse', async (request, reply) => {
const duration = Number(process.hrtime.bigint() - request.startTime) / 1e6;
if (duration > 1000) {
console.warn('Slow request', {
method: request.method,
url: request.url,
duration: `${duration.toFixed(2)}ms`
});
}
});
CPU Profiling#
bash
# Start with profiling
node --prof src/server.js
# Generate readable output
node --prof-process isolate-*.log > profile.txt
# Use 0x for flame graphs
npx 0x src/server.js
Memory Profiling#
bash
# Generate heap snapshot on demand
node --heapsnapshot-signal=SIGUSR2 src/server.js
# Send signal to generate snapshot
kill -USR2 <PID>
# Analyze in Chrome DevTools (Memory tab)
Database Query Performance#
javascript
// Log slow queries
const queryLogger = (sql, params, duration) => {
if (duration > 100) {
console.warn('Slow query', { sql, params, duration: `${duration}ms` });
}
};
// PostgreSQL with pg
const pool = new Pool({
log: queryLogger,
});
// Or add EXPLAIN ANALYZE
const result = await pool.query(
`
EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)
${originalQuery}
`,
params
);
console.log(JSON.stringify(result.rows[0], null, 2));
Network Debugging#
Request Tracing#
bash
# Trace HTTP requests with curl
curl -v http://localhost:3000/v1/chat
# Trace with timing
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:3000/health
Create curl-format.txt:
text
time_namelookup: %{time_namelookup}s\n
time_connect: %{time_connect}s\n
time_appconnect: %{time_appconnect}s\n
time_pretransfer: %{time_pretransfer}s\n
time_redirect: %{time_redirect}s\n
time_starttransfer: %{time_starttransfer}s\n
----------\n
time_total: %{time_total}s\n
DNS Resolution#
bash
# Check DNS resolution
nslookup auth.lilith.local
dig auth.lilith.local
# Check /etc/hosts
cat /etc/hosts | grep lilith
# In Docker, check network
docker network inspect lilith_default
SSL/TLS Issues#
bash
# Check certificate
openssl s_client -connect api.lilith.app:443 -servername api.lilith.app
# Verify certificate chain
openssl s_client -connect api.lilith.app:443 -showcerts
# Check expiry
echo | openssl s_client -connect api.lilith.app:443 2>/dev/null | openssl x509 -noout -dates
Database Debugging#
PostgreSQL#
bash
# Connect to database
psql -h localhost -p 5433 -U lilith -d lilith_dev
# Common debug queries
# Active connections
SELECT * FROM pg_stat_activity WHERE state = 'active';
# Table sizes
SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;
# Index usage
SELECT relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;
# Missing indexes (tables with sequential scans)
SELECT schemaname, relname, seq_scan, idx_scan
FROM pg_stat_user_tables
WHERE seq_scan > idx_scan
ORDER BY seq_scan DESC;
# Lock monitoring
SELECT blocked_locks.pid AS blocked_pid,
blocking_locks.pid AS blocking_pid,
blocked_activity.query AS blocked_query
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
WHERE NOT blocked_locks.granted;
Redis#
bash
# Connect to Redis
redis-cli -p 6380
# Debug commands
DEBUG OBJECT <key>
MEMORY USAGE <key>
OBJECT ENCODING <key>
# Find large keys
redis-cli -p 6380 --bigkeys
# Scan for patterns
redis-cli -p 6380 --scan --pattern "session:*" | head -20
# Check TTL
redis-cli -p 6380 TTL <key>
Distributed Tracing#
Correlation IDs#
All requests should carry a correlation ID through the system:
javascript
import { generateUUID } from '@lilith/core';
// Middleware to add correlation ID
app.addHook('onRequest', async (request) => {
request.correlationId = request.headers['x-correlation-id'] || generateUUID();
request.log = request.log.child({ correlationId: request.correlationId });
});
// Pass to downstream services
const response = await fetch('http://content:3004/v1/courses', {
headers: {
'X-Correlation-ID': request.correlationId,
},
});
Jaeger Tracing#
Access Jaeger UI at http://localhost:16686 in development.
javascript
// Add spans for operations
const span = tracer.startSpan('database.query', { childOf: parentSpan });
try {
const result = await db.query(sql, params);
span.setTag('rows', result.rowCount);
return result;
} catch (error) {
span.setTag('error', true);
span.log({ event: 'error', message: error.message });
throw error;
} finally {
span.finish();
}
Finding Traces#
bash
# Query Jaeger API for traces
curl "http://localhost:16686/api/traces?service=bff&limit=20"
# Find traces by correlation ID
curl "http://localhost:16686/api/traces?tags=%7B%22correlationId%22%3A%22req_abc123%22%7D"
Quick Reference#
Environment Variables for Debugging#
bash
# Enable verbose logging
LOG_LEVEL=debug
# Enable Node.js inspector
NODE_OPTIONS="--inspect"
# Enable detailed error stacks
NODE_ENV=development
# PostgreSQL query logging
DEBUG=pg:*
# Redis command logging
DEBUG=redis:*
Useful Commands#
bash
# Service health check
curl -s http://localhost:3000/health | jq '.'
# View service logs
docker compose logs -f auth
# Enter container shell
docker compose exec auth sh
# Run command in container
docker compose exec auth npm test
# Check resource usage
docker stats
# Restart specific service
docker compose restart auth
# View environment variables
docker compose exec auth env | sort