# Logging Infrastructure

Yemaya uses the ELK Stack (Elasticsearch, Logstash, Kibana) for centralized
logging, log processing, and visualization.

## Overview

The logging infrastructure provides:

- **Elasticsearch**: Log storage and search
- **Logstash**: Log processing and transformation
- **Kibana**: Log visualization and dashboards
- **Structured logging**: JSON format with correlation IDs
- **Log retention**: Automated lifecycle management
- **Sensitive data masking**: Automatic PII redaction

## Quick Start

### Start the Logging Stack

```bash
# Start with logging profile
docker compose --profile logging up -d

# Wait for initialization
docker compose logs -f elasticsearch-init
```

### Access Services

| Service       | URL                   | Credentials                  |
| ------------- | --------------------- | ---------------------------- |
| Elasticsearch | http://localhost:9200 | elastic / yemaya_elastic_dev |
| Kibana        | http://localhost:5601 | elastic / yemaya_elastic_dev |
| Logstash      | http://localhost:9600 | (monitoring API)             |

### Send a Test Log

```bash
# Send via TCP
echo '{"level":"info","service":"test","message":"Hello Yemaya!"}' | nc localhost 5000

# Or via HTTP
curl -X POST http://localhost:5000 \
  -H "Content-Type: application/json" \
  -d '{"level":"info","service":"test","message":"Hello Yemaya!"}'
```

## Architecture

```
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   API       │────▶│  Logstash   │────▶│Elasticsearch│
│   Workers   │     │  (process)  │     │  (store)    │
│   Services  │     └─────────────┘     └─────────────┘
└─────────────┘                               │
                                              ▼
                                        ┌─────────────┐
                                        │   Kibana    │
                                        │ (visualize) │
                                        └─────────────┘
```

## Log Format

### Standard Log Structure

```json
{
  "@timestamp": "2024-12-27T12:00:00.000Z",
  "level": "info",
  "service": "api",
  "environment": "production",
  "message": "Request processed successfully",
  "correlation_id": "abc-123-def-456",
  "request_id": "req-789",
  "user_id": "user-123",
  "duration_ms": 45,
  "http": {
    "method": "GET",
    "path": "/api/users",
    "status_code": 200
  }
}
```

### Log Levels

| Level   | Usage                                   |
| ------- | --------------------------------------- |
| `fatal` | Application crash, unrecoverable errors |
| `error` | Errors requiring investigation          |
| `warn`  | Potential issues, degraded performance  |
| `info`  | Normal operations, key events           |
| `debug` | Detailed debugging information          |
| `trace` | Very detailed tracing                   |

## Sending Logs

### From Node.js Applications

```typescript
import pino from 'pino';
import { createWriteStream } from 'net';

// Create logger with TCP transport
const logger = pino({
  level: 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
  base: {
    service: 'api',
    environment: process.env.NODE_ENV,
  },
  transport: {
    targets: [
      // Console output
      { target: 'pino-pretty', level: 'debug' },
      // Logstash TCP
      {
        target: 'pino-socket',
        options: {
          address: 'localhost',
          port: 5000,
          mode: 'tcp',
        },
        level: 'info',
      },
    ],
  },
});

// Add correlation ID to all logs
const childLogger = logger.child({
  correlation_id: 'abc-123',
  request_id: 'req-456',
});

childLogger.info({ user_id: 'user-789' }, 'User logged in');
```

### From HTTP (Direct to Elasticsearch)

```bash
curl -X POST "http://localhost:9200/yemaya-logs/_doc" \
  -H "Content-Type: application/json" \
  -u elastic:yemaya_elastic_dev \
  -d '{
    "@timestamp": "2024-12-27T12:00:00.000Z",
    "level": "info",
    "service": "external",
    "message": "External system log"
  }'
```

## Index Structure

### Indices

| Index Pattern      | Purpose             | Retention |
| ------------------ | ------------------- | --------- |
| `yemaya-logs-*`    | Application logs    | 30 days   |
| `yemaya-audit-*`   | Security audit logs | 90 days   |
| `yemaya-metrics-*` | Application metrics | 7 days    |

### Write Aliases

Always use these aliases for writing:

- `yemaya-logs` - Application logs
- `yemaya-audit` - Audit logs
- `yemaya-metrics` - Metrics

### Index Lifecycle

```
HOT (0-1 day)     →  WARM (7 days)    →  COLD (14 days)   →  DELETE (30 days)
- Active writes      - Read-only          - Compressed         - Removed
- Fast queries       - Merged segments    - Slow queries
```

## Logstash Pipeline

### Input Ports

| Port | Protocol | Format     |
| ---- | -------- | ---------- |
| 5000 | TCP      | JSON Lines |
| 5044 | TCP      | Beats      |
| 8080 | HTTP     | JSON       |

### Processing

1. **Parse JSON** - Extract structured fields
2. **Normalize levels** - Standardize log levels
3. **Add metadata** - Environment, timestamps
4. **Mask sensitive data** - Redact PII
5. **Fingerprint** - Deduplicate logs
6. **Route** - Send to Elasticsearch

### Sensitive Data Masking

Automatically masked patterns:

- Email addresses → `[EMAIL_REDACTED]`
- JWT tokens → `[JWT_REDACTED]`
- API keys → `[API_KEY_REDACTED]`
- Credit cards → `[CARD_REDACTED]`

## Kibana Dashboards

### Default Dashboards

1. **Overview** - High-level metrics
2. **Errors** - Error trends and details
3. **API Performance** - Request latencies
4. **User Activity** - User actions audit

### Creating Dashboards

1. Go to Kibana → Analytics → Dashboard
2. Create new visualization
3. Select `yemaya-logs` index pattern
4. Build visualization
5. Add to dashboard

## Production Setup

### Security Configuration

```bash
# Run security setup
ELASTIC_PASSWORD=your-password ./infrastructure/elasticsearch/scripts/setup-security.sh all
```

### TLS Certificates

```bash
# Generate certificates
./infrastructure/elasticsearch/scripts/setup-security.sh certs

# Copy to appropriate locations
cp /tmp/elasticsearch-certs/* infrastructure/elasticsearch/certs/
```

### Environment Variables

| Variable                | Description              | Default            |
| ----------------------- | ------------------------ | ------------------ |
| `ELASTIC_PASSWORD`      | Elastic user password    | yemaya_elastic_dev |
| `KIBANA_PASSWORD`       | Kibana system password   | yemaya_kibana_dev  |
| `KIBANA_ENCRYPTION_KEY` | Saved objects encryption | (generated)        |

### Multi-Node Cluster

For production, run multiple Elasticsearch nodes:

```yaml
# docker-compose.prod.yml
services:
  es01:
    image: elasticsearch:8.11.3
    environment:
      - node.name=es01
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
  es02:
    # Similar config...
  es03:
    # Similar config...
```

## Monitoring

### Cluster Health

```bash
# Check cluster health
curl -u elastic:password http://localhost:9200/_cluster/health?pretty

# Check indices
curl -u elastic:password http://localhost:9200/_cat/indices?v

# Check shards
curl -u elastic:password http://localhost:9200/_cat/shards?v
```

### Useful Queries

```bash
# Recent errors
curl -u elastic:password -X GET "http://localhost:9200/yemaya-logs/_search" \
  -H "Content-Type: application/json" \
  -d '{"query":{"bool":{"must":[{"term":{"level":"error"}},{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}'

# Logs by service
curl -u elastic:password -X GET "http://localhost:9200/yemaya-logs/_search" \
  -H "Content-Type: application/json" \
  -d '{"aggs":{"by_service":{"terms":{"field":"service"}}}}'
```

## Troubleshooting

### Elasticsearch Won't Start

1. Check memory settings:

   ```bash
   docker exec yemaya-elasticsearch cat /proc/meminfo
   ```

2. Check disk space:

   ```bash
   docker exec yemaya-elasticsearch df -h
   ```

3. Check logs:
   ```bash
   docker logs yemaya-elasticsearch
   ```

### Logs Not Appearing

1. Check Logstash:

   ```bash
   docker logs yemaya-logstash
   curl http://localhost:9600/_node/stats
   ```

2. Test connectivity:

   ```bash
   nc -zv localhost 5000
   ```

3. Check pipeline:
   ```bash
   curl http://localhost:9600/_node/pipelines
   ```

### High Memory Usage

1. Reduce heap size in `jvm.options`
2. Enable index lifecycle management
3. Delete old indices:
   ```bash
   curl -X DELETE -u elastic:password "http://localhost:9200/yemaya-logs-2024.01.*"
   ```

## Alerting

Yemaya includes pre-configured Elasticsearch Watchers for log-based alerting.

### Included Alerts

| Alert                    | Description               | Trigger                           |
| ------------------------ | ------------------------- | --------------------------------- |
| **error-rate-alert**     | High error rate detection | >50 errors in 5 minutes           |
| **slow-requests-alert**  | Slow request monitoring   | >10 requests over 5s in 5 minutes |
| **service-health-alert** | Service health check      | >10% error rate per service       |
| **auth-failure-alert**   | Authentication failures   | >20 auth failures in 5 minutes    |
| **error-spike-alert**    | Anomaly detection         | 3x baseline error rate            |
| **server-error-alert**   | HTTP 5xx errors           | >5 5xx errors in 2 minutes        |

### Setting Up Alerts

```bash
# Start the logging stack with watchers
docker compose --profile logging up -d

# The watcher-init container automatically registers all alerts
docker compose logs watcher-init
```

### Configuring Slack Notifications

Set the `SLACK_WEBHOOK` environment variable:

```bash
export SLACK_WEBHOOK="T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
docker compose --profile logging up -d
```

### Managing Watchers

```bash
# List all watchers
curl -u elastic:password http://localhost:9200/_watcher/watch/*

# View specific watcher
curl -u elastic:password http://localhost:9200/_watcher/watch/error-rate-alert

# Execute watcher manually (for testing)
curl -X POST -u elastic:password \
  http://localhost:9200/_watcher/watch/error-rate-alert/_execute

# Deactivate a watcher
curl -X PUT -u elastic:password \
  http://localhost:9200/_watcher/watch/error-rate-alert/_deactivate

# Activate a watcher
curl -X PUT -u elastic:password \
  http://localhost:9200/_watcher/watch/error-rate-alert/_activate

# Delete a watcher
curl -X DELETE -u elastic:password \
  http://localhost:9200/_watcher/watch/error-rate-alert
```

### Creating Custom Alerts

1. Create a JSON watcher definition in `infrastructure/elasticsearch/watchers/`
2. Restart the watcher-init container:
   ```bash
   docker compose --profile logging restart watcher-init
   ```

Example custom watcher:

```json
{
  "trigger": { "schedule": { "interval": "5m" } },
  "input": {
    "search": {
      "request": {
        "indices": ["yemaya-logs-*"],
        "body": {
          "query": {
            "bool": {
              "must": [
                { "term": { "level": "error" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": { "ctx.payload.hits.total.value": { "gt": 10 } }
  },
  "actions": {
    "log_alert": {
      "logging": { "text": "Custom alert triggered!" }
    }
  }
}
```

### Alert Actions

Watchers support multiple action types:

- **logging** - Log to Elasticsearch
- **webhook** - HTTP POST (Slack, PagerDuty, etc.)
- **email** - Send email notifications
- **index** - Index alert to Elasticsearch

## Kibana Dashboards

### Included Dashboards

The following dashboards are automatically imported:

1. **Yemaya Logs Overview** - Main monitoring dashboard
   - Errors over time
   - Logs by level and service
   - Response time histogram
   - HTTP status codes
   - Top endpoints
   - Recent errors

2. **Service Health Dashboard** - Service-specific monitoring
   - Service log volume
   - Error rates by service
   - Response times by service
   - Log levels heatmap

### Saved Searches

Pre-configured searches for quick access:

- **All Logs** - Full log view with key columns
- **Error Logs** - Error and fatal logs with stack traces
- **HTTP Requests** - HTTP request details
- **Slow Requests** - Requests over 1 second
- **Auth Events** - Authentication and authorization logs
- **Server Errors (5xx)** - HTTP 500-level errors

### Accessing Dashboards

1. Open Kibana: http://localhost:5601
2. Login with `elastic` / `yemaya_elastic_dev`
3. Go to Analytics → Dashboard
4. Select a dashboard

## Log Sampling

For high-volume services, Yemaya supports log sampling at both the application
and infrastructure level.

### Why Sampling?

- Reduces storage costs for log data
- Prevents infrastructure overload during traffic spikes
- Maintains statistically significant samples for analysis
- Keeps critical logs (errors, warnings) at 100%

### Application-Level Sampling

Use the `@yemaya/logging` package samplers:

```typescript
import {
  createAdaptiveSampler,
  createSamplingTransport,
  createTcpTransport,
} from '@yemaya/logging';

// Create an adaptive sampler
const sampler = createAdaptiveSampler({
  rate: 0.5, // Base 50% sample rate
  targetLogsPerSecond: 100, // Target 100 logs/sec
  minRate: 0.01, // Minimum 1%
  maxRate: 1.0, // Maximum 100%
  adjustInterval: 5000, // Adjust every 5 seconds
  alwaysLogLevel: 'warn', // Always log warn and above
});

// Wrap transport with sampler
const transport = createSamplingTransport(
  createTcpTransport({
    host: 'localhost',
    port: 5000,
  }),
  sampler
);
```

### Available Samplers

| Sampler                   | Description                      | Use Case              |
| ------------------------- | -------------------------------- | --------------------- |
| **FixedRateSampler**      | Constant sample rate             | Predictable volume    |
| **AdaptiveSampler**       | Adjusts rate based on volume     | Variable traffic      |
| **PrioritySampler**       | Different rates by priority      | Mixed importance logs |
| **ConsistentHashSampler** | Consistent sampling by attribute | User/request tracing  |

### Fixed Rate Example

```typescript
import { createFixedRateSampler } from '@yemaya/logging';

const sampler = createFixedRateSampler({
  rate: 0.1, // 10% sample rate
  alwaysLogLevel: 'error', // Always log errors
  alwaysLogServices: ['auth'], // Always log auth service
  alwaysLogPaths: ['/api/payment'], // Always log payment
});
```

### Priority-Based Example

```typescript
import { createPrioritySampler } from '@yemaya/logging';

const sampler = createPrioritySampler({
  rate: 0.5,
  priorityField: 'importance',
  priorities: {
    100: 1.0, // Priority 100 = 100% sampled
    50: 0.5, // Priority 50 = 50% sampled
    10: 0.1, // Priority 10 = 10% sampled
  },
  defaultPriority: 50,
});
```

### Consistent Hash Example

```typescript
import { createConsistentHashSampler } from '@yemaya/logging';

// Sample consistently by user ID
const sampler = createConsistentHashSampler({
  rate: 0.1, // 10% of users
  hashField: 'userId', // Hash on userId
  fallbackField: 'requestId', // Fallback to requestId
  alwaysLogLevel: 'error',
});
```

### Logstash-Level Sampling

The Logstash pipeline applies additional sampling:

| Log Type                 | Sample Rate | Reason                             |
| ------------------------ | ----------- | ---------------------------------- |
| `error`, `warn`, `fatal` | 100%        | Always critical                    |
| `info`                   | 100%        | Normal operations                  |
| `debug`                  | 10%         | Reduce debug noise                 |
| `trace`                  | 1%          | Minimal trace logs                 |
| High-volume endpoints    | 50%         | `/api/v1/assets`, `/api/v1/search` |
| Health checks            | 0%          | Dropped entirely                   |

### Viewing Sampling Statistics

```bash
# In application logs, check _sampling metadata
curl -u elastic:password -X GET \
  "http://localhost:9200/yemaya-logs/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "aggs": {
      "by_sample_reason": {
        "terms": { "field": "_sampling.reason" }
      }
    }
  }'
```

### Configuring Sample Rates

Override default sampling in Logstash:

```ruby
# In yemaya.conf, modify the ruby blocks
ruby {
  code => "
    sample_rate = ENV['DEBUG_SAMPLE_RATE'] || 0.10
    # ... rest of sampling logic
  "
}
```

Or set environment variables:

```bash
export DEBUG_SAMPLE_RATE=0.25
export TRACE_SAMPLE_RATE=0.05
docker compose --profile logging restart logstash
```

## Related Documentation

- [Environment Variables](./ENVIRONMENT_VARIABLES.md)
- [Monitoring](./MONITORING.md)
- [Docker Compose](../docker-compose.yml)
