Yemaya uses the ELK Stack (Elasticsearch, Logstash, Kibana) for centralized logging, log processing, and visualization.
Overview#
The logging infrastructure provides:
- Elasticsearch: Log storage and search
- Logstash: Log processing and transformation
- Kibana: Log visualization and dashboards
- Structured logging: JSON format with correlation IDs
- Log retention: Automated lifecycle management
- Sensitive data masking: Automatic PII redaction
Quick Start#
Start the Logging Stack#
# Start with logging profile
docker compose --profile logging up -d
# Wait for initialization
docker compose logs -f elasticsearch-init
Access Services#
| Service | URL | Credentials |
|---|---|---|
| Elasticsearch | http://localhost:9200 | elastic / yemaya_elastic_dev |
| Kibana | http://localhost:5601 | elastic / yemaya_elastic_dev |
| Logstash | http://localhost:9600 | (monitoring API) |
Send a Test Log#
# Send via TCP
echo '{"level":"info","service":"test","message":"Hello Yemaya!"}' | nc localhost 5000
# Or via HTTP
curl -X POST http://localhost:5000 \
-H "Content-Type: application/json" \
-d '{"level":"info","service":"test","message":"Hello Yemaya!"}'
Architecture#
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ API │────▶│ Logstash │────▶│Elasticsearch│
│ Workers │ │ (process) │ │ (store) │
│ Services │ └─────────────┘ └─────────────┘
└─────────────┘ │
▼
┌─────────────┐
│ Kibana │
│ (visualize) │
└─────────────┘
Log Format#
Standard Log Structure#
{
"@timestamp": "2024-12-27T12:00:00.000Z",
"level": "info",
"service": "api",
"environment": "production",
"message": "Request processed successfully",
"correlation_id": "abc-123-def-456",
"request_id": "req-789",
"user_id": "user-123",
"duration_ms": 45,
"http": {
"method": "GET",
"path": "/api/users",
"status_code": 200
}
}
Log Levels#
| Level | Usage |
|---|---|
fatal |
Application crash, unrecoverable errors |
error |
Errors requiring investigation |
warn |
Potential issues, degraded performance |
info |
Normal operations, key events |
debug |
Detailed debugging information |
trace |
Very detailed tracing |
Sending Logs#
From Node.js Applications#
import pino from 'pino';
import { createWriteStream } from 'net';
// Create logger with TCP transport
const logger = pino({
level: 'info',
formatters: {
level: (label) => ({ level: label }),
},
base: {
service: 'api',
environment: process.env.NODE_ENV,
},
transport: {
targets: [
// Console output
{ target: 'pino-pretty', level: 'debug' },
// Logstash TCP
{
target: 'pino-socket',
options: {
address: 'localhost',
port: 5000,
mode: 'tcp',
},
level: 'info',
},
],
},
});
// Add correlation ID to all logs
const childLogger = logger.child({
correlation_id: 'abc-123',
request_id: 'req-456',
});
childLogger.info({ user_id: 'user-789' }, 'User logged in');
From HTTP (Direct to Elasticsearch)#
curl -X POST "http://localhost:9200/yemaya-logs/_doc" \
-H "Content-Type: application/json" \
-u elastic:yemaya_elastic_dev \
-d '{
"@timestamp": "2024-12-27T12:00:00.000Z",
"level": "info",
"service": "external",
"message": "External system log"
}'
Index Structure#
Indices#
| Index Pattern | Purpose | Retention |
|---|---|---|
yemaya-logs-* |
Application logs | 30 days |
yemaya-audit-* |
Security audit logs | 90 days |
yemaya-metrics-* |
Application metrics | 7 days |
Write Aliases#
Always use these aliases for writing:
yemaya-logs- Application logsyemaya-audit- Audit logsyemaya-metrics- Metrics
Index Lifecycle#
HOT (0-1 day) → WARM (7 days) → COLD (14 days) → DELETE (30 days)
- Active writes - Read-only - Compressed - Removed
- Fast queries - Merged segments - Slow queries
Logstash Pipeline#
Input Ports#
| Port | Protocol | Format |
|---|---|---|
| 5000 | TCP | JSON Lines |
| 5044 | TCP | Beats |
| 8080 | HTTP | JSON |
Processing#
- Parse JSON - Extract structured fields
- Normalize levels - Standardize log levels
- Add metadata - Environment, timestamps
- Mask sensitive data - Redact PII
- Fingerprint - Deduplicate logs
- Route - Send to Elasticsearch
Sensitive Data Masking#
Automatically masked patterns:
- Email addresses →
[EMAIL_REDACTED] - JWT tokens →
[JWT_REDACTED] - API keys →
[API_KEY_REDACTED] - Credit cards →
[CARD_REDACTED]
Kibana Dashboards#
Default Dashboards#
- Overview - High-level metrics
- Errors - Error trends and details
- API Performance - Request latencies
- User Activity - User actions audit
Creating Dashboards#
- Go to Kibana → Analytics → Dashboard
- Create new visualization
- Select
yemaya-logsindex pattern - Build visualization
- Add to dashboard
Production Setup#
Security Configuration#
# Run security setup
ELASTIC_PASSWORD=your-password ./infrastructure/elasticsearch/scripts/setup-security.sh all
TLS Certificates#
# Generate certificates
./infrastructure/elasticsearch/scripts/setup-security.sh certs
# Copy to appropriate locations
cp /tmp/elasticsearch-certs/* infrastructure/elasticsearch/certs/
Environment Variables#
| Variable | Description | Default |
|---|---|---|
ELASTIC_PASSWORD |
Elastic user password | yemaya_elastic_dev |
KIBANA_PASSWORD |
Kibana system password | yemaya_kibana_dev |
KIBANA_ENCRYPTION_KEY |
Saved objects encryption | (generated) |
Multi-Node Cluster#
For production, run multiple Elasticsearch nodes:
# docker-compose.prod.yml
services:
es01:
image: elasticsearch:8.11.3
environment:
- node.name=es01
- cluster.initial_master_nodes=es01,es02,es03
- discovery.seed_hosts=es02,es03
es02:
# Similar config...
es03:
# Similar config...
Monitoring#
Cluster Health#
# Check cluster health
curl -u elastic:password http://localhost:9200/_cluster/health?pretty
# Check indices
curl -u elastic:password http://localhost:9200/_cat/indices?v
# Check shards
curl -u elastic:password http://localhost:9200/_cat/shards?v
Useful Queries#
# Recent errors
curl -u elastic:password -X GET "http://localhost:9200/yemaya-logs/_search" \
-H "Content-Type: application/json" \
-d '{"query":{"bool":{"must":[{"term":{"level":"error"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}'
# Logs by service
curl -u elastic:password -X GET "http://localhost:9200/yemaya-logs/_search" \
-H "Content-Type: application/json" \
-d '{"aggs":{"by_service":{"terms":{"field":"service"}}}}'
Troubleshooting#
Elasticsearch Won't Start#
-
Check memory settings:
bashdocker exec yemaya-elasticsearch cat /proc/meminfo -
Check disk space:
bashdocker exec yemaya-elasticsearch df -h -
Check logs:
bashdocker logs yemaya-elasticsearch
Logs Not Appearing#
-
Check Logstash:
bashdocker logs yemaya-logstash curl http://localhost:9600/_node/stats -
Test connectivity:
bashnc -zv localhost 5000 -
Check pipeline:
bashcurl http://localhost:9600/_node/pipelines
High Memory Usage#
- Reduce heap size in
jvm.options - Enable index lifecycle management
- Delete old indices:bash
curl -X DELETE -u elastic:password "http://localhost:9200/yemaya-logs-2024.01.*"
Alerting#
Yemaya includes pre-configured Elasticsearch Watchers for log-based alerting.
Included Alerts#
| Alert | Description | Trigger |
|---|---|---|
| error-rate-alert | High error rate detection | >50 errors in 5 minutes |
| slow-requests-alert | Slow request monitoring | >10 requests over 5s in 5 minutes |
| service-health-alert | Service health check | >10% error rate per service |
| auth-failure-alert | Authentication failures | >20 auth failures in 5 minutes |
| error-spike-alert | Anomaly detection | 3x baseline error rate |
| server-error-alert | HTTP 5xx errors | >5 5xx errors in 2 minutes |
Setting Up Alerts#
# Start the logging stack with watchers
docker compose --profile logging up -d
# The watcher-init container automatically registers all alerts
docker compose logs watcher-init
Configuring Slack Notifications#
Set the SLACK_WEBHOOK environment variable:
export SLACK_WEBHOOK="T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
docker compose --profile logging up -d
Managing Watchers#
# List all watchers
curl -u elastic:password http://localhost:9200/_watcher/watch/*
# View specific watcher
curl -u elastic:password http://localhost:9200/_watcher/watch/error-rate-alert
# Execute watcher manually (for testing)
curl -X POST -u elastic:password \
http://localhost:9200/_watcher/watch/error-rate-alert/_execute
# Deactivate a watcher
curl -X PUT -u elastic:password \
http://localhost:9200/_watcher/watch/error-rate-alert/_deactivate
# Activate a watcher
curl -X PUT -u elastic:password \
http://localhost:9200/_watcher/watch/error-rate-alert/_activate
# Delete a watcher
curl -X DELETE -u elastic:password \
http://localhost:9200/_watcher/watch/error-rate-alert
Creating Custom Alerts#
- Create a JSON watcher definition in
infrastructure/elasticsearch/watchers/ - Restart the watcher-init container:bash
docker compose --profile logging restart watcher-init
Example custom watcher:
{
"trigger": { "schedule": { "interval": "5m" } },
"input": {
"search": {
"request": {
"indices": ["yemaya-logs-*"],
"body": {
"query": {
"bool": {
"must": [
{ "term": { "level": "error" } },
{ "range": { "@timestamp": { "gte": "now-5m" } } }
]
}
}
}
}
}
},
"condition": {
"compare": { "ctx.payload.hits.total.value": { "gt": 10 } }
},
"actions": {
"log_alert": {
"logging": { "text": "Custom alert triggered!" }
}
}
}
Alert Actions#
Watchers support multiple action types:
- logging - Log to Elasticsearch
- webhook - HTTP POST (Slack, PagerDuty, etc.)
- email - Send email notifications
- index - Index alert to Elasticsearch
Kibana Dashboards#
Included Dashboards#
The following dashboards are automatically imported:
-
Yemaya Logs Overview - Main monitoring dashboard
- Errors over time
- Logs by level and service
- Response time histogram
- HTTP status codes
- Top endpoints
- Recent errors
-
Service Health Dashboard - Service-specific monitoring
- Service log volume
- Error rates by service
- Response times by service
- Log levels heatmap
Saved Searches#
Pre-configured searches for quick access:
- All Logs - Full log view with key columns
- Error Logs - Error and fatal logs with stack traces
- HTTP Requests - HTTP request details
- Slow Requests - Requests over 1 second
- Auth Events - Authentication and authorization logs
- Server Errors (5xx) - HTTP 500-level errors
Accessing Dashboards#
- Open Kibana: http://localhost:5601
- Login with
elastic/yemaya_elastic_dev - Go to Analytics → Dashboard
- Select a dashboard
Log Sampling#
For high-volume services, Yemaya supports log sampling at both the application and infrastructure level.
Why Sampling?#
- Reduces storage costs for log data
- Prevents infrastructure overload during traffic spikes
- Maintains statistically significant samples for analysis
- Keeps critical logs (errors, warnings) at 100%
Application-Level Sampling#
Use the @yemaya/logging package samplers:
import {
createAdaptiveSampler,
createSamplingTransport,
createTcpTransport,
} from '@yemaya/logging';
// Create an adaptive sampler
const sampler = createAdaptiveSampler({
rate: 0.5, // Base 50% sample rate
targetLogsPerSecond: 100, // Target 100 logs/sec
minRate: 0.01, // Minimum 1%
maxRate: 1.0, // Maximum 100%
adjustInterval: 5000, // Adjust every 5 seconds
alwaysLogLevel: 'warn', // Always log warn and above
});
// Wrap transport with sampler
const transport = createSamplingTransport(
createTcpTransport({
host: 'localhost',
port: 5000,
}),
sampler
);
Available Samplers#
| Sampler | Description | Use Case |
|---|---|---|
| FixedRateSampler | Constant sample rate | Predictable volume |
| AdaptiveSampler | Adjusts rate based on volume | Variable traffic |
| PrioritySampler | Different rates by priority | Mixed importance logs |
| ConsistentHashSampler | Consistent sampling by attribute | User/request tracing |
Fixed Rate Example#
import { createFixedRateSampler } from '@yemaya/logging';
const sampler = createFixedRateSampler({
rate: 0.1, // 10% sample rate
alwaysLogLevel: 'error', // Always log errors
alwaysLogServices: ['auth'], // Always log auth service
alwaysLogPaths: ['/api/payment'], // Always log payment
});
Priority-Based Example#
import { createPrioritySampler } from '@yemaya/logging';
const sampler = createPrioritySampler({
rate: 0.5,
priorityField: 'importance',
priorities: {
100: 1.0, // Priority 100 = 100% sampled
50: 0.5, // Priority 50 = 50% sampled
10: 0.1, // Priority 10 = 10% sampled
},
defaultPriority: 50,
});
Consistent Hash Example#
import { createConsistentHashSampler } from '@yemaya/logging';
// Sample consistently by user ID
const sampler = createConsistentHashSampler({
rate: 0.1, // 10% of users
hashField: 'userId', // Hash on userId
fallbackField: 'requestId', // Fallback to requestId
alwaysLogLevel: 'error',
});
Logstash-Level Sampling#
The Logstash pipeline applies additional sampling:
| Log Type | Sample Rate | Reason |
|---|---|---|
error, warn, fatal |
100% | Always critical |
info |
100% | Normal operations |
debug |
10% | Reduce debug noise |
trace |
1% | Minimal trace logs |
| High-volume endpoints | 50% | /api/v1/assets, /api/v1/search |
| Health checks | 0% | Dropped entirely |
Viewing Sampling Statistics#
# In application logs, check _sampling metadata
curl -u elastic:password -X GET \
"http://localhost:9200/yemaya-logs/_search" \
-H "Content-Type: application/json" \
-d '{
"aggs": {
"by_sample_reason": {
"terms": { "field": "_sampling.reason" }
}
}
}'
Configuring Sample Rates#
Override default sampling in Logstash:
# In yemaya.conf, modify the ruby blocks
ruby {
code => "
sample_rate = ENV['DEBUG_SAMPLE_RATE'] || 0.10
# ... rest of sampling logic
"
}
Or set environment variables:
export DEBUG_SAMPLE_RATE=0.25
export TRACE_SAMPLE_RATE=0.05
docker compose --profile logging restart logstash