Eve library

Logging Standards and Best Practices

Guide

Revision 1

On this page

Overview#

All Lilith platform services must use LilithLogger for structured, consistent logging with OpenTelemetry integration, PII redaction, and centralized log aggregation.

Current Status#

Services with LilithLogger Integration (12/40):

  • ✅ BFF Service
  • ✅ Auth Service
  • ✅ Content Service
  • ✅ Conversation Service
  • ✅ Moderation Service
  • ✅ Auth-Orchestrator Service
  • ✅ Error-Handler Service
  • ✅ Group-Meditation Service
  • ✅ Offline Service
  • ✅ Progress-Sync Service
  • ✅ Real-Time-Sync Service
  • ✅ Review Service

Services Requiring LilithLogger Integration (28/40): See "Migration Guide" section below for remaining services.

Standard Pattern#

1. Import LilithLogger#

typescript
import { LilithLogger } from '../../lib/lilith-logger.js';

2. Initialize Logger#

typescript
export async function buildServer(
  opts: BuildServerOptions = {}
): Promise<FastifyInstance> {
  const SERVICE =
    opts.serviceName || process.env['SERVICE_NAME'] || 'service-name';
  const environment =
    opts.environment || process.env['ENVIRONMENT'] || 'development';

  // Initialize structured logger
  const logger = new LilithLogger({
    service: SERVICE,
    version: process.env['SERVICE_VERSION'] || '1.0.0',
    environment,
    outputs: ['console', 'elasticsearch'],
  });

  // ... rest of buildServer
}

3. Integrate with Fastify#

typescript
const app = Fastify({
  logger: {
    level: 'info',
    stream: {
      write: (msg: string): void => {
        try {
          const logData = JSON.parse(msg);
          logger.log(logData.level, logData.msg, {
            metadata: logData,
          });
        } catch (_e) {
          logger.info(msg.trim());
        }
      },
    },
  },
});

// Attach structured logger to app for easy access
app.decorate('structuredLogger', logger);

4. Add Response Time Helper#

typescript
function computeResponseTime(
  request: any = {},
  reply: any = {}
): number | undefined {
  if (
    reply &&
    typeof reply.elapsedTime === 'number' &&
    Number.isFinite(reply.elapsedTime)
  ) {
    return reply.elapsedTime;
  }
  const startTime = request && request.startTime;
  if (typeof startTime === 'number' && Number.isFinite(startTime)) {
    const diff = Date.now() - startTime;
    return diff >= 0 ? diff : 0;
  }
  return undefined;
}

5. Add Request Tracking Hooks#

typescript
// Track request start time
app.addHook('onRequest', async (request, _reply) => {
  (request as any).startTime = Date.now();
});

// Log HTTP requests
app.addHook('onResponse', async (request, reply) => {
  const responseTime = computeResponseTime(request, reply);
  logger.logHttpRequest(request, reply, responseTime);
});

Logger API#

Core Methods#

typescript
// Basic logging levels
logger.trace(message: string, context?: object): void
logger.debug(message: string, context?: object): void
logger.info(message: string, context?: object): void
logger.warn(message: string, context?: object): void
logger.error(message: string, context?: object): void
logger.fatal(message: string, context?: object): void

// Generic log method
logger.log(level: string, message: string, context?: object): void

// HTTP request logging with automatic metrics
logger.logHttpRequest(request: FastifyRequest, reply: FastifyReply, responseTime?: number): void

// Performance logging
logger.logPerformance(
  operation: string,
  duration: number,
  category: string,
  metadata?: object
): void

Usage Examples#

typescript
// Simple logging
logger.info('User logged in successfully', {
  userId: user.id,
  email: user.email, // Will be PII-redacted automatically
});

// Error logging with stack traces
logger.error('Database query failed', {
  query: query,
  error: error.message,
  stack: error.stack,
});

// Performance logging
const startTime = Date.now();
await performOperation();
logger.logPerformance('database_query', Date.now() - startTime, 'database', {
  table: 'users',
  operation: 'SELECT',
});

Features#

1. Automatic PII Redaction#

LilithLogger automatically redacts sensitive information:

  • Email addresses → ***@***.***
  • User IDs → hashed values
  • IP addresses → anonymized
  • User agents → sanitized
typescript
// Input
logger.info('User action', {
  email: 'user@example.com',
  ip: '192.168.1.1'
});

// Logged (PII redacted)
{
  message: 'User action',
  email: '***@***.***',
  ip: '192.168.***'
}

2. OpenTelemetry Integration#

Automatically includes trace context in logs:

  • trace_id: Distributed trace identifier
  • span_id: Current span identifier
  • parent_span_id: Parent span identifier

3. Elasticsearch Integration#

Logs are automatically sent to Elasticsearch for:

  • Centralized log aggregation
  • Advanced search and filtering
  • Log retention and archival
  • Analytics and dashboards

4. Performance Thresholds#

Automatically logs slow operations:

typescript
// Configure thresholds
const logger = new LoggerLib({
  service: 'my-service',
  performance: {
    threshold_ms: 1000, // Log operations > 1 second
    enable_slow_query_logging: true,
  },
});

5. Sampling and Batching#

For high-volume services:

typescript
const logger = new LoggerLib({
  service: 'high-volume-service',
  sampling: {
    enable: true,
    rate: 0.1, // Sample 10% of logs
  },
  batching: {
    enable: true,
    batch_size: 100,
    flush_interval_ms: 5000,
  },
});

Best Practices#

1. Structured Logging#

✅ DO:

typescript
logger.info('Payment processed', {
  userId: payment.userId,
  amount: payment.amount,
  currency: payment.currency,
  paymentId: payment.id,
});

❌ DON'T:

typescript
logger.info(`Payment processed for user ${userId} amount ${amount}`);

2. Contextual Information#

Always include relevant context:

typescript
logger.error('API request failed', {
  method: request.method,
  url: request.url,
  statusCode: error.statusCode,
  error: error.message,
  userId: request.user?.id,
  correlationId: request.headers['x-correlation-id'],
});

3. Log Levels#

  • TRACE: Extremely detailed, rarely used
  • DEBUG: Detailed debugging information
  • INFO: Normal operational messages (default)
  • WARN: Warning messages (potential issues)
  • ERROR: Error conditions (needs attention)
  • FATAL: Critical failures (system unusable)

4. Performance Logging#

Log slow operations automatically:

typescript
const startTime = Date.now();
const result = await database.query(sql);
const duration = Date.now() - startTime;

if (duration > 1000) {
  logger.logPerformance('slow_query', duration, 'database', {
    sql: sql.substring(0, 100), // Truncate for logging
    rows: result.rows.length,
  });
}

5. Security#

Never log sensitive data explicitly:

typescript
// ❌ DON'T
logger.info('User authenticated', {
  password: user.password, // NEVER!
  creditCard: payment.cardNumber, // NEVER!
});

// ✅ DO (rely on automatic PII redaction)
logger.info('User authenticated', {
  userId: user.id,
  email: user.email, // Will be redacted
});

Testing Logging#

Integration Tests#

typescript
import { test, expect } from '@playwright/test';

test('Service logs HTTP requests with proper format', async ({ request }) => {
  // Make request
  const response = await request.get('http://localhost:8080/v1/health');

  // Verify response
  expect(response.status()).toBe(200);

  // Verify logs (check console output or Elasticsearch)
  // This requires log capture mechanism in tests
});

Unit Tests#

typescript
describe('LilithLogger', () => {
  it('should redact PII from logs', () => {
    const logger = new LilithLogger({ service: 'test' });
    const logSpy = jest.spyOn(console, 'log');

    logger.info('Test message', {
      email: 'user@example.com',
      ip: '192.168.1.1',
    });

    expect(logSpy).toHaveBeenCalledWith(
      expect.objectContaining({
        email: '***@***.***',
        ip: expect.stringContaining('***'),
      })
    );
  });
});

Migration Guide#

Services Requiring Integration#

  1. AI Service
  2. Media Service
  3. Audio-Handoff Service
  4. Community-Translation Service
  5. Content-Licensing Service
  6. Vectordb Service
  7. User-Preferences Service
  8. Notification Service
  9. Observability Service
  10. Catalog Service
  11. Ingestion Service
  12. STT Service
  13. TTS Service
  14. WebRTC Service
  15. RAG Service
  16. Knowledge Service
  17. Settlement Service
  18. Payment-Orchestrator Service
  19. Partner-API Service
  20. Language-Detection Service
  21. Data-Governance Service
  22. Consent-Management Service
  23. Anonymization Service
  24. Voice-Pipeline Service
  25. Token-Verification Service
  26. Multi-Region-Resilience Service
  27. Safety-Automation Service
  28. Rights-Management Service

Migration Steps#

For each service:

  1. Add import: import { LilithLogger as LoggerLib } from '../../lib/index.js';
  2. Initialize logger: Follow "Standard Pattern" section above
  3. Update Fastify config: Integrate logger with Fastify stream
  4. Add response time helper: Copy computeResponseTime function
  5. Add request hooks: onRequest and onResponse hooks
  6. Replace console.log: Remove all console.log/error/warn with logger.info/error/warn
  7. Test integration: Verify logs appear in console and Elasticsearch
  8. Update tests: Add logging verification to integration tests

Automation Script#

For automated migration of remaining services:

bash
#!/bin/bash
# scripts/integrate-lilith-logger.sh

SERVICE_NAME=$1
SERVICE_PATH="services/${SERVICE_NAME}/src/app.ts"

if [ ! -f "$SERVICE_PATH" ]; then
  echo "Service not found: $SERVICE_PATH"
  exit 1
fi

# Backup original
cp "$SERVICE_PATH" "${SERVICE_PATH}.backup"

# Apply transformation (use sed/awk or Node.js script)
node scripts/integrate-logger.js "$SERVICE_NAME"

echo "Integrated LilithLogger into ${SERVICE_NAME}"
echo "Please review changes and run tests"

Monitoring and Observability#

Elasticsearch Queries#

json
// Find all ERROR logs from last hour
{
  "query": {
    "bool": {
      "must": [
        { "match": { "level": "ERROR" }},
        { "range": { "timestamp": { "gte": "now-1h" }}}
      ]
    }
  }
}

// Find slow database queries
{
  "query": {
    "bool": {
      "must": [
        { "match": { "category": "database" }},
        { "range": { "duration_ms": { "gte": 1000 }}}
      ]
    }
  }
}

Grafana Dashboards#

Create dashboards for:

  • Request volume by service
  • Error rate by service
  • P50/P95/P99 latency
  • Slow query detection
  • PII redaction metrics

Troubleshooting#

Logger Not Working#

  1. Check environment variables:
bash
export LOG_LEVEL=info
export ELASTICSEARCH_URL=http://localhost:9200
  1. Verify Elasticsearch connection:
typescript
logger.info('Test log'); // Should appear in console and Elasticsearch
  1. Check for initialization errors:
typescript
try {
  const logger = new LilithLogger({ service: 'test' });
} catch (error) {
  console.error('Logger initialization failed:', error);
}

Missing Logs#

  1. Check sampling configuration (might be filtering logs)
  2. Verify log level (might be too high)
  3. Check Elasticsearch indices: GET /_cat/indices/lilith-logs-*

Performance Issues#

  1. Enable batching for high-volume services
  2. Increase batch size: batch_size: 1000
  3. Reduce flush interval: flush_interval_ms: 10000
  4. Enable sampling: sampling: { rate: 0.1 }

References#

Support#

For questions or issues with logging:

  1. Check existing service implementations for examples
  2. Review LilithLogger source code
  3. Test in development environment first
  4. Verify Elasticsearch connectivity

Last Updated: October 10, 2025 Status: 12/40 services integrated, 28 remaining Priority: HIGH - Critical for production observability