Eve library

Logging Infrastructure

Guide

Revision 1

On this page

Yemaya uses the ELK Stack (Elasticsearch, Logstash, Kibana) for centralized logging, log processing, and visualization.

Overview#

The logging infrastructure provides:

  • Elasticsearch: Log storage and search
  • Logstash: Log processing and transformation
  • Kibana: Log visualization and dashboards
  • Structured logging: JSON format with correlation IDs
  • Log retention: Automated lifecycle management
  • Sensitive data masking: Automatic PII redaction

Quick Start#

Start the Logging Stack#

bash
# Start with logging profile
docker compose --profile logging up -d

# Wait for initialization
docker compose logs -f elasticsearch-init

Access Services#

Service URL Credentials
Elasticsearch http://localhost:9200 elastic / yemaya_elastic_dev
Kibana http://localhost:5601 elastic / yemaya_elastic_dev
Logstash http://localhost:9600 (monitoring API)

Send a Test Log#

bash
# Send via TCP
echo '{"level":"info","service":"test","message":"Hello Yemaya!"}' | nc localhost 5000

# Or via HTTP
curl -X POST http://localhost:5000 \
  -H "Content-Type: application/json" \
  -d '{"level":"info","service":"test","message":"Hello Yemaya!"}'

Architecture#

text
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   API       │────▶│  Logstash   │────▶│Elasticsearch│
│   Workers   │     │  (process)  │     │  (store)    │
│   Services  │     └─────────────┘     └─────────────┘
└─────────────┘                               │
                                              ▼
                                        ┌─────────────┐
                                        │   Kibana    │
                                        │ (visualize) │
                                        └─────────────┘

Log Format#

Standard Log Structure#

json
{
  "@timestamp": "2024-12-27T12:00:00.000Z",
  "level": "info",
  "service": "api",
  "environment": "production",
  "message": "Request processed successfully",
  "correlation_id": "abc-123-def-456",
  "request_id": "req-789",
  "user_id": "user-123",
  "duration_ms": 45,
  "http": {
    "method": "GET",
    "path": "/api/users",
    "status_code": 200
  }
}

Log Levels#

Level Usage
fatal Application crash, unrecoverable errors
error Errors requiring investigation
warn Potential issues, degraded performance
info Normal operations, key events
debug Detailed debugging information
trace Very detailed tracing

Sending Logs#

From Node.js Applications#

typescript
import pino from 'pino';
import { createWriteStream } from 'net';

// Create logger with TCP transport
const logger = pino({
  level: 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
  base: {
    service: 'api',
    environment: process.env.NODE_ENV,
  },
  transport: {
    targets: [
      // Console output
      { target: 'pino-pretty', level: 'debug' },
      // Logstash TCP
      {
        target: 'pino-socket',
        options: {
          address: 'localhost',
          port: 5000,
          mode: 'tcp',
        },
        level: 'info',
      },
    ],
  },
});

// Add correlation ID to all logs
const childLogger = logger.child({
  correlation_id: 'abc-123',
  request_id: 'req-456',
});

childLogger.info({ user_id: 'user-789' }, 'User logged in');

From HTTP (Direct to Elasticsearch)#

bash
curl -X POST "http://localhost:9200/yemaya-logs/_doc" \
  -H "Content-Type: application/json" \
  -u elastic:yemaya_elastic_dev \
  -d '{
    "@timestamp": "2024-12-27T12:00:00.000Z",
    "level": "info",
    "service": "external",
    "message": "External system log"
  }'

Index Structure#

Indices#

Index Pattern Purpose Retention
yemaya-logs-* Application logs 30 days
yemaya-audit-* Security audit logs 90 days
yemaya-metrics-* Application metrics 7 days

Write Aliases#

Always use these aliases for writing:

  • yemaya-logs - Application logs
  • yemaya-audit - Audit logs
  • yemaya-metrics - Metrics

Index Lifecycle#

text
HOT (0-1 day)     →  WARM (7 days)    →  COLD (14 days)   →  DELETE (30 days)
- Active writes      - Read-only          - Compressed         - Removed
- Fast queries       - Merged segments    - Slow queries

Logstash Pipeline#

Input Ports#

Port Protocol Format
5000 TCP JSON Lines
5044 TCP Beats
8080 HTTP JSON

Processing#

  1. Parse JSON - Extract structured fields
  2. Normalize levels - Standardize log levels
  3. Add metadata - Environment, timestamps
  4. Mask sensitive data - Redact PII
  5. Fingerprint - Deduplicate logs
  6. Route - Send to Elasticsearch

Sensitive Data Masking#

Automatically masked patterns:

  • Email addresses → [EMAIL_REDACTED]
  • JWT tokens → [JWT_REDACTED]
  • API keys → [API_KEY_REDACTED]
  • Credit cards → [CARD_REDACTED]

Kibana Dashboards#

Default Dashboards#

  1. Overview - High-level metrics
  2. Errors - Error trends and details
  3. API Performance - Request latencies
  4. User Activity - User actions audit

Creating Dashboards#

  1. Go to Kibana → Analytics → Dashboard
  2. Create new visualization
  3. Select yemaya-logs index pattern
  4. Build visualization
  5. Add to dashboard

Production Setup#

Security Configuration#

bash
# Run security setup
ELASTIC_PASSWORD=your-password ./infrastructure/elasticsearch/scripts/setup-security.sh all

TLS Certificates#

bash
# Generate certificates
./infrastructure/elasticsearch/scripts/setup-security.sh certs

# Copy to appropriate locations
cp /tmp/elasticsearch-certs/* infrastructure/elasticsearch/certs/

Environment Variables#

Variable Description Default
ELASTIC_PASSWORD Elastic user password yemaya_elastic_dev
KIBANA_PASSWORD Kibana system password yemaya_kibana_dev
KIBANA_ENCRYPTION_KEY Saved objects encryption (generated)

Multi-Node Cluster#

For production, run multiple Elasticsearch nodes:

yaml
# docker-compose.prod.yml
services:
  es01:
    image: elasticsearch:8.11.3
    environment:
      - node.name=es01
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
  es02:
    # Similar config...
  es03:
    # Similar config...

Monitoring#

Cluster Health#

bash
# Check cluster health
curl -u elastic:password http://localhost:9200/_cluster/health?pretty

# Check indices
curl -u elastic:password http://localhost:9200/_cat/indices?v

# Check shards
curl -u elastic:password http://localhost:9200/_cat/shards?v

Useful Queries#

bash
# Recent errors
curl -u elastic:password -X GET "http://localhost:9200/yemaya-logs/_search" \
  -H "Content-Type: application/json" \
  -d '{"query":{"bool":{"must":[{"term":{"level":"error"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}'

# Logs by service
curl -u elastic:password -X GET "http://localhost:9200/yemaya-logs/_search" \
  -H "Content-Type: application/json" \
  -d '{"aggs":{"by_service":{"terms":{"field":"service"}}}}'

Troubleshooting#

Elasticsearch Won't Start#

  1. Check memory settings:

    bash
    docker exec yemaya-elasticsearch cat /proc/meminfo
    
  2. Check disk space:

    bash
    docker exec yemaya-elasticsearch df -h
    
  3. Check logs:

    bash
    docker logs yemaya-elasticsearch
    

Logs Not Appearing#

  1. Check Logstash:

    bash
    docker logs yemaya-logstash
    curl http://localhost:9600/_node/stats
    
  2. Test connectivity:

    bash
    nc -zv localhost 5000
    
  3. Check pipeline:

    bash
    curl http://localhost:9600/_node/pipelines
    

High Memory Usage#

  1. Reduce heap size in jvm.options
  2. Enable index lifecycle management
  3. Delete old indices:
    bash
    curl -X DELETE -u elastic:password "http://localhost:9200/yemaya-logs-2024.01.*"
    

Alerting#

Yemaya includes pre-configured Elasticsearch Watchers for log-based alerting.

Included Alerts#

Alert Description Trigger
error-rate-alert High error rate detection >50 errors in 5 minutes
slow-requests-alert Slow request monitoring >10 requests over 5s in 5 minutes
service-health-alert Service health check >10% error rate per service
auth-failure-alert Authentication failures >20 auth failures in 5 minutes
error-spike-alert Anomaly detection 3x baseline error rate
server-error-alert HTTP 5xx errors >5 5xx errors in 2 minutes

Setting Up Alerts#

bash
# Start the logging stack with watchers
docker compose --profile logging up -d

# The watcher-init container automatically registers all alerts
docker compose logs watcher-init

Configuring Slack Notifications#

Set the SLACK_WEBHOOK environment variable:

bash
export SLACK_WEBHOOK="T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
docker compose --profile logging up -d

Managing Watchers#

bash
# List all watchers
curl -u elastic:password http://localhost:9200/_watcher/watch/*

# View specific watcher
curl -u elastic:password http://localhost:9200/_watcher/watch/error-rate-alert

# Execute watcher manually (for testing)
curl -X POST -u elastic:password \
  http://localhost:9200/_watcher/watch/error-rate-alert/_execute

# Deactivate a watcher
curl -X PUT -u elastic:password \
  http://localhost:9200/_watcher/watch/error-rate-alert/_deactivate

# Activate a watcher
curl -X PUT -u elastic:password \
  http://localhost:9200/_watcher/watch/error-rate-alert/_activate

# Delete a watcher
curl -X DELETE -u elastic:password \
  http://localhost:9200/_watcher/watch/error-rate-alert

Creating Custom Alerts#

  1. Create a JSON watcher definition in infrastructure/elasticsearch/watchers/
  2. Restart the watcher-init container:
    bash
    docker compose --profile logging restart watcher-init
    

Example custom watcher:

json
{
  "trigger": { "schedule": { "interval": "5m" } },
  "input": {
    "search": {
      "request": {
        "indices": ["yemaya-logs-*"],
        "body": {
          "query": {
            "bool": {
              "must": [
                { "term": { "level": "error" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": { "ctx.payload.hits.total.value": { "gt": 10 } }
  },
  "actions": {
    "log_alert": {
      "logging": { "text": "Custom alert triggered!" }
    }
  }
}

Alert Actions#

Watchers support multiple action types:

  • logging - Log to Elasticsearch
  • webhook - HTTP POST (Slack, PagerDuty, etc.)
  • email - Send email notifications
  • index - Index alert to Elasticsearch

Kibana Dashboards#

Included Dashboards#

The following dashboards are automatically imported:

  1. Yemaya Logs Overview - Main monitoring dashboard

    • Errors over time
    • Logs by level and service
    • Response time histogram
    • HTTP status codes
    • Top endpoints
    • Recent errors
  2. Service Health Dashboard - Service-specific monitoring

    • Service log volume
    • Error rates by service
    • Response times by service
    • Log levels heatmap

Saved Searches#

Pre-configured searches for quick access:

  • All Logs - Full log view with key columns
  • Error Logs - Error and fatal logs with stack traces
  • HTTP Requests - HTTP request details
  • Slow Requests - Requests over 1 second
  • Auth Events - Authentication and authorization logs
  • Server Errors (5xx) - HTTP 500-level errors

Accessing Dashboards#

  1. Open Kibana: http://localhost:5601
  2. Login with elastic / yemaya_elastic_dev
  3. Go to Analytics → Dashboard
  4. Select a dashboard

Log Sampling#

For high-volume services, Yemaya supports log sampling at both the application and infrastructure level.

Why Sampling?#

  • Reduces storage costs for log data
  • Prevents infrastructure overload during traffic spikes
  • Maintains statistically significant samples for analysis
  • Keeps critical logs (errors, warnings) at 100%

Application-Level Sampling#

Use the @yemaya/logging package samplers:

typescript
import {
  createAdaptiveSampler,
  createSamplingTransport,
  createTcpTransport,
} from '@yemaya/logging';

// Create an adaptive sampler
const sampler = createAdaptiveSampler({
  rate: 0.5, // Base 50% sample rate
  targetLogsPerSecond: 100, // Target 100 logs/sec
  minRate: 0.01, // Minimum 1%
  maxRate: 1.0, // Maximum 100%
  adjustInterval: 5000, // Adjust every 5 seconds
  alwaysLogLevel: 'warn', // Always log warn and above
});

// Wrap transport with sampler
const transport = createSamplingTransport(
  createTcpTransport({
    host: 'localhost',
    port: 5000,
  }),
  sampler
);

Available Samplers#

Sampler Description Use Case
FixedRateSampler Constant sample rate Predictable volume
AdaptiveSampler Adjusts rate based on volume Variable traffic
PrioritySampler Different rates by priority Mixed importance logs
ConsistentHashSampler Consistent sampling by attribute User/request tracing

Fixed Rate Example#

typescript
import { createFixedRateSampler } from '@yemaya/logging';

const sampler = createFixedRateSampler({
  rate: 0.1, // 10% sample rate
  alwaysLogLevel: 'error', // Always log errors
  alwaysLogServices: ['auth'], // Always log auth service
  alwaysLogPaths: ['/api/payment'], // Always log payment
});

Priority-Based Example#

typescript
import { createPrioritySampler } from '@yemaya/logging';

const sampler = createPrioritySampler({
  rate: 0.5,
  priorityField: 'importance',
  priorities: {
    100: 1.0, // Priority 100 = 100% sampled
    50: 0.5, // Priority 50 = 50% sampled
    10: 0.1, // Priority 10 = 10% sampled
  },
  defaultPriority: 50,
});

Consistent Hash Example#

typescript
import { createConsistentHashSampler } from '@yemaya/logging';

// Sample consistently by user ID
const sampler = createConsistentHashSampler({
  rate: 0.1, // 10% of users
  hashField: 'userId', // Hash on userId
  fallbackField: 'requestId', // Fallback to requestId
  alwaysLogLevel: 'error',
});

Logstash-Level Sampling#

The Logstash pipeline applies additional sampling:

Log Type Sample Rate Reason
error, warn, fatal 100% Always critical
info 100% Normal operations
debug 10% Reduce debug noise
trace 1% Minimal trace logs
High-volume endpoints 50% /api/v1/assets, /api/v1/search
Health checks 0% Dropped entirely

Viewing Sampling Statistics#

bash
# In application logs, check _sampling metadata
curl -u elastic:password -X GET \
  "http://localhost:9200/yemaya-logs/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "aggs": {
      "by_sample_reason": {
        "terms": { "field": "_sampling.reason" }
      }
    }
  }'

Configuring Sample Rates#

Override default sampling in Logstash:

ruby
# In yemaya.conf, modify the ruby blocks
ruby {
  code => "
    sample_rate = ENV['DEBUG_SAMPLE_RATE'] || 0.10
    # ... rest of sampling logic
  "
}

Or set environment variables:

bash
export DEBUG_SAMPLE_RATE=0.25
export TRACE_SAMPLE_RATE=0.05
docker compose --profile logging restart logstash