Overview#
All Lilith platform services must use LilithLogger for structured, consistent
logging with OpenTelemetry integration, PII redaction, and centralized log
aggregation.
Current Status#
Services with LilithLogger Integration (12/40):
- ✅ BFF Service
- ✅ Auth Service
- ✅ Content Service
- ✅ Conversation Service
- ✅ Moderation Service
- ✅ Auth-Orchestrator Service
- ✅ Error-Handler Service
- ✅ Group-Meditation Service
- ✅ Offline Service
- ✅ Progress-Sync Service
- ✅ Real-Time-Sync Service
- ✅ Review Service
Services Requiring LilithLogger Integration (28/40): See "Migration Guide" section below for remaining services.
Standard Pattern#
1. Import LilithLogger#
import { LilithLogger } from '../../lib/lilith-logger.js';
2. Initialize Logger#
export async function buildServer(
opts: BuildServerOptions = {}
): Promise<FastifyInstance> {
const SERVICE =
opts.serviceName || process.env['SERVICE_NAME'] || 'service-name';
const environment =
opts.environment || process.env['ENVIRONMENT'] || 'development';
// Initialize structured logger
const logger = new LilithLogger({
service: SERVICE,
version: process.env['SERVICE_VERSION'] || '1.0.0',
environment,
outputs: ['console', 'elasticsearch'],
});
// ... rest of buildServer
}
3. Integrate with Fastify#
const app = Fastify({
logger: {
level: 'info',
stream: {
write: (msg: string): void => {
try {
const logData = JSON.parse(msg);
logger.log(logData.level, logData.msg, {
metadata: logData,
});
} catch (_e) {
logger.info(msg.trim());
}
},
},
},
});
// Attach structured logger to app for easy access
app.decorate('structuredLogger', logger);
4. Add Response Time Helper#
function computeResponseTime(
request: any = {},
reply: any = {}
): number | undefined {
if (
reply &&
typeof reply.elapsedTime === 'number' &&
Number.isFinite(reply.elapsedTime)
) {
return reply.elapsedTime;
}
const startTime = request && request.startTime;
if (typeof startTime === 'number' && Number.isFinite(startTime)) {
const diff = Date.now() - startTime;
return diff >= 0 ? diff : 0;
}
return undefined;
}
5. Add Request Tracking Hooks#
// Track request start time
app.addHook('onRequest', async (request, _reply) => {
(request as any).startTime = Date.now();
});
// Log HTTP requests
app.addHook('onResponse', async (request, reply) => {
const responseTime = computeResponseTime(request, reply);
logger.logHttpRequest(request, reply, responseTime);
});
Logger API#
Core Methods#
// Basic logging levels
logger.trace(message: string, context?: object): void
logger.debug(message: string, context?: object): void
logger.info(message: string, context?: object): void
logger.warn(message: string, context?: object): void
logger.error(message: string, context?: object): void
logger.fatal(message: string, context?: object): void
// Generic log method
logger.log(level: string, message: string, context?: object): void
// HTTP request logging with automatic metrics
logger.logHttpRequest(request: FastifyRequest, reply: FastifyReply, responseTime?: number): void
// Performance logging
logger.logPerformance(
operation: string,
duration: number,
category: string,
metadata?: object
): void
Usage Examples#
// Simple logging
logger.info('User logged in successfully', {
userId: user.id,
email: user.email, // Will be PII-redacted automatically
});
// Error logging with stack traces
logger.error('Database query failed', {
query: query,
error: error.message,
stack: error.stack,
});
// Performance logging
const startTime = Date.now();
await performOperation();
logger.logPerformance('database_query', Date.now() - startTime, 'database', {
table: 'users',
operation: 'SELECT',
});
Features#
1. Automatic PII Redaction#
LilithLogger automatically redacts sensitive information:
- Email addresses →
***@***.*** - User IDs → hashed values
- IP addresses → anonymized
- User agents → sanitized
// Input
logger.info('User action', {
email: 'user@example.com',
ip: '192.168.1.1'
});
// Logged (PII redacted)
{
message: 'User action',
email: '***@***.***',
ip: '192.168.***'
}
2. OpenTelemetry Integration#
Automatically includes trace context in logs:
trace_id: Distributed trace identifierspan_id: Current span identifierparent_span_id: Parent span identifier
3. Elasticsearch Integration#
Logs are automatically sent to Elasticsearch for:
- Centralized log aggregation
- Advanced search and filtering
- Log retention and archival
- Analytics and dashboards
4. Performance Thresholds#
Automatically logs slow operations:
// Configure thresholds
const logger = new LoggerLib({
service: 'my-service',
performance: {
threshold_ms: 1000, // Log operations > 1 second
enable_slow_query_logging: true,
},
});
5. Sampling and Batching#
For high-volume services:
const logger = new LoggerLib({
service: 'high-volume-service',
sampling: {
enable: true,
rate: 0.1, // Sample 10% of logs
},
batching: {
enable: true,
batch_size: 100,
flush_interval_ms: 5000,
},
});
Best Practices#
1. Structured Logging#
✅ DO:
logger.info('Payment processed', {
userId: payment.userId,
amount: payment.amount,
currency: payment.currency,
paymentId: payment.id,
});
❌ DON'T:
logger.info(`Payment processed for user ${userId} amount ${amount}`);
2. Contextual Information#
Always include relevant context:
logger.error('API request failed', {
method: request.method,
url: request.url,
statusCode: error.statusCode,
error: error.message,
userId: request.user?.id,
correlationId: request.headers['x-correlation-id'],
});
3. Log Levels#
- TRACE: Extremely detailed, rarely used
- DEBUG: Detailed debugging information
- INFO: Normal operational messages (default)
- WARN: Warning messages (potential issues)
- ERROR: Error conditions (needs attention)
- FATAL: Critical failures (system unusable)
4. Performance Logging#
Log slow operations automatically:
const startTime = Date.now();
const result = await database.query(sql);
const duration = Date.now() - startTime;
if (duration > 1000) {
logger.logPerformance('slow_query', duration, 'database', {
sql: sql.substring(0, 100), // Truncate for logging
rows: result.rows.length,
});
}
5. Security#
Never log sensitive data explicitly:
// ❌ DON'T
logger.info('User authenticated', {
password: user.password, // NEVER!
creditCard: payment.cardNumber, // NEVER!
});
// ✅ DO (rely on automatic PII redaction)
logger.info('User authenticated', {
userId: user.id,
email: user.email, // Will be redacted
});
Testing Logging#
Integration Tests#
import { test, expect } from '@playwright/test';
test('Service logs HTTP requests with proper format', async ({ request }) => {
// Make request
const response = await request.get('http://localhost:8080/v1/health');
// Verify response
expect(response.status()).toBe(200);
// Verify logs (check console output or Elasticsearch)
// This requires log capture mechanism in tests
});
Unit Tests#
describe('LilithLogger', () => {
it('should redact PII from logs', () => {
const logger = new LilithLogger({ service: 'test' });
const logSpy = jest.spyOn(console, 'log');
logger.info('Test message', {
email: 'user@example.com',
ip: '192.168.1.1',
});
expect(logSpy).toHaveBeenCalledWith(
expect.objectContaining({
email: '***@***.***',
ip: expect.stringContaining('***'),
})
);
});
});
Migration Guide#
Services Requiring Integration#
- AI Service
- Media Service
- Audio-Handoff Service
- Community-Translation Service
- Content-Licensing Service
- Vectordb Service
- User-Preferences Service
- Notification Service
- Observability Service
- Catalog Service
- Ingestion Service
- STT Service
- TTS Service
- WebRTC Service
- RAG Service
- Knowledge Service
- Settlement Service
- Payment-Orchestrator Service
- Partner-API Service
- Language-Detection Service
- Data-Governance Service
- Consent-Management Service
- Anonymization Service
- Voice-Pipeline Service
- Token-Verification Service
- Multi-Region-Resilience Service
- Safety-Automation Service
- Rights-Management Service
Migration Steps#
For each service:
- Add import:
import { LilithLogger as LoggerLib } from '../../lib/index.js'; - Initialize logger: Follow "Standard Pattern" section above
- Update Fastify config: Integrate logger with Fastify stream
- Add response time helper: Copy
computeResponseTimefunction - Add request hooks:
onRequestandonResponsehooks - Replace console.log: Remove all
console.log/error/warnwithlogger.info/error/warn - Test integration: Verify logs appear in console and Elasticsearch
- Update tests: Add logging verification to integration tests
Automation Script#
For automated migration of remaining services:
#!/bin/bash
# scripts/integrate-lilith-logger.sh
SERVICE_NAME=$1
SERVICE_PATH="services/${SERVICE_NAME}/src/app.ts"
if [ ! -f "$SERVICE_PATH" ]; then
echo "Service not found: $SERVICE_PATH"
exit 1
fi
# Backup original
cp "$SERVICE_PATH" "${SERVICE_PATH}.backup"
# Apply transformation (use sed/awk or Node.js script)
node scripts/integrate-logger.js "$SERVICE_NAME"
echo "Integrated LilithLogger into ${SERVICE_NAME}"
echo "Please review changes and run tests"
Monitoring and Observability#
Elasticsearch Queries#
// Find all ERROR logs from last hour
{
"query": {
"bool": {
"must": [
{ "match": { "level": "ERROR" }},
{ "range": { "timestamp": { "gte": "now-1h" }}}
]
}
}
}
// Find slow database queries
{
"query": {
"bool": {
"must": [
{ "match": { "category": "database" }},
{ "range": { "duration_ms": { "gte": 1000 }}}
]
}
}
}
Grafana Dashboards#
Create dashboards for:
- Request volume by service
- Error rate by service
- P50/P95/P99 latency
- Slow query detection
- PII redaction metrics
Troubleshooting#
Logger Not Working#
- Check environment variables:
export LOG_LEVEL=info
export ELASTICSEARCH_URL=http://localhost:9200
- Verify Elasticsearch connection:
logger.info('Test log'); // Should appear in console and Elasticsearch
- Check for initialization errors:
try {
const logger = new LilithLogger({ service: 'test' });
} catch (error) {
console.error('Logger initialization failed:', error);
}
Missing Logs#
- Check sampling configuration (might be filtering logs)
- Verify log level (might be too high)
- Check Elasticsearch indices:
GET /_cat/indices/lilith-logs-*
Performance Issues#
- Enable batching for high-volume services
- Increase batch size:
batch_size: 1000 - Reduce flush interval:
flush_interval_ms: 10000 - Enable sampling:
sampling: { rate: 0.1 }
References#
- LilithLogger Implementation
- OpenTelemetry Tracing
- Elasticsearch Integration
- Example Service: Conversation
- Example Service: Moderation
- Example Service: BFF
Support#
For questions or issues with logging:
- Check existing service implementations for examples
- Review LilithLogger source code
- Test in development environment first
- Verify Elasticsearch connectivity
Last Updated: October 10, 2025 Status: 12/40 services integrated, 28 remaining Priority: HIGH - Critical for production observability