Eve library

Distributed Tracing Standards

Guide

Revision 1

On this page

Comprehensive guide for implementing distributed tracing across the Lilith platform.

Table of Contents#

Overview#

Distributed tracing provides visibility into request flows across microservices, enabling performance analysis and debugging.

Tracing Stack#

Component Technology Purpose
SDK @lilith/tracing Application instrumentation
Collector OpenTelemetry Collector Trace aggregation
Backend Tempo Trace storage
Visualization Grafana Trace exploration

Trace Flow#

text
┌─────────────────────────────────────────────────────────────────────────┐
│                          Request Flow                                    │
│                                                                          │
│  Client ──▶ BFF ──▶ Auth ──▶ Content ──▶ AI                            │
│    │         │        │         │        │                              │
│    ▼         ▼        ▼         ▼        ▼                              │
│  Span 1   Span 2   Span 3   Span 4   Span 5                            │
│  (root)   (child)  (child)  (child)  (child)                           │
│    │         │        │         │        │                              │
│    └─────────┴────────┴─────────┴────────┘                              │
│                        │                                                 │
│                        ▼                                                 │
│              ┌─────────────────────┐                                    │
│              │  OpenTelemetry      │                                    │
│              │  Collector          │                                    │
│              └─────────────────────┘                                    │
│                        │                                                 │
│                        ▼                                                 │
│              ┌─────────────────────┐                                    │
│              │       Tempo         │                                    │
│              │  (Trace Storage)    │                                    │
│              └─────────────────────┘                                    │
│                        │                                                 │
│                        ▼                                                 │
│              ┌─────────────────────┐                                    │
│              │      Grafana        │                                    │
│              │  (Visualization)    │                                    │
│              └─────────────────────┘                                    │
└─────────────────────────────────────────────────────────────────────────┘

Tracing Architecture#

W3C Trace Context#

All services use W3C Trace Context for distributed tracing:

text
traceparent: 00-{trace-id}-{span-id}-{trace-flags}

Example:
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
                 └────────trace-id────────────┘ └───span-id──────┘ └flags

Core Components#

typescript
// packages/tracing/src/index.ts

export interface SpanContext {
  traceId: string; // 32 hex chars
  spanId: string; // 16 hex chars
  traceFlags: number; // 1 = sampled
  traceState?: string;
}

export interface Span {
  name: string;
  context: SpanContext;
  parentSpanId?: string;
  startTime: number;
  endTime?: number;
  attributes: SpanAttributes;
  events: SpanEvent[];
  status: SpanStatus;
}

Implementation Guide#

Basic Setup#

typescript
import { createTracer } from '@lilith/tracing';

// Create a tracer for your service
const tracer = createTracer({
  serviceName: 'auth-service',
  serviceVersion: '1.0.0',
  enabled: true,
  sampleRate: 1.0,
});

// Use in request handlers
app.get('/api/users/:id', async (request, reply) => {
  return tracer.startSpan('getUser', async (span) => {
    span.setAttribute('user.id', request.params.id);

    const user = await userService.findById(request.params.id);

    span.setAttribute('user.found', !!user);
    return user;
  });
});

Fastify Integration#

typescript
import Fastify from 'fastify';
import {
  createTracer,
  extractTraceContext,
  injectTraceContext,
} from '@lilith/tracing';

const app = Fastify();
const tracer = createTracer('my-service');

// Add tracing hook
app.addHook('onRequest', async (request, reply) => {
  const parentContext = extractTraceContext(request.headers);

  request.traceContext = parentContext;
  request.span = tracer.startSpan(
    `${request.method} ${request.url}`,
    parentContext
  );

  request.span.setAttribute('http.method', request.method);
  request.span.setAttribute('http.url', request.url);
  request.span.setAttribute(
    'http.target',
    request.routeOptions?.url || request.url
  );
});

app.addHook('onResponse', async (request, reply) => {
  request.span.setAttribute('http.status_code', reply.statusCode);
  request.span.end();
});

app.addHook('onError', async (request, reply, error) => {
  request.span.setStatus({
    code: SpanStatusCode.ERROR,
    message: error.message,
  });
  request.span.addEvent('exception', {
    'exception.type': error.name,
    'exception.message': error.message,
    'exception.stacktrace': error.stack,
  });
});

Service-to-Service Calls#

typescript
import { injectTraceContext } from '@lilith/tracing';

async function callDownstreamService(tracer, parentSpan) {
  return tracer.startSpan('call-content-service', async (span) => {
    const headers = {};
    injectTraceContext(span.context, headers);

    const response = await fetch('http://content-service/api/data', {
      headers: {
        ...headers,
        'Content-Type': 'application/json',
      },
    });

    span.setAttribute('http.status_code', response.status);
    return response.json();
  });
}

Instrumentation Standards#

Required Attributes#

HTTP Spans#

Attribute Type Description
http.method string HTTP method (GET, POST, etc.)
http.url string Full URL
http.target string Path and query string
http.status_code number Response status code
http.host string Host header
http.scheme string http or https
typescript
span.setAttributes({
  'http.method': 'GET',
  'http.url': 'https://api.lilith.app/v1/users/123',
  'http.target': '/v1/users/123',
  'http.status_code': 200,
  'http.host': 'api.lilith.app',
  'http.scheme': 'https',
});

Database Spans#

Attribute Type Description
db.system string Database type (postgresql, redis)
db.name string Database name
db.statement string Query (sanitized)
db.operation string Operation type (query, insert)
typescript
span.setAttributes({
  'db.system': 'postgresql',
  'db.name': 'lilith_auth',
  'db.statement': 'SELECT * FROM users WHERE id = $1',
  'db.operation': 'query',
});

gRPC Spans#

Attribute Type Description
rpc.system string 'grpc'
rpc.service string Service name
rpc.method string Method name
rpc.grpc.status_code number gRPC status code

Messaging Spans#

Attribute Type Description
messaging.system string kafka, rabbitmq, etc.
messaging.destination string Queue/topic name
messaging.operation string send, receive, process

Custom Business Attributes#

typescript
// User context
span.setAttribute('user.id', userId);
span.setAttribute('user.role', userRole);

// Content context
span.setAttribute('content.id', contentId);
span.setAttribute('content.type', 'meditation');

// AI context
span.setAttribute('ai.model', 'gpt-4');
span.setAttribute('ai.tokens_used', tokenCount);

Span Naming Conventions#

HTTP Endpoints#

text
{HTTP_METHOD} {PATH_TEMPLATE}

Examples:
GET /v1/users/{userId}
POST /v1/content/courses
DELETE /v1/sessions/{sessionId}

Database Operations#

text
{OPERATION} {TABLE/COLLECTION}

Examples:
SELECT users
INSERT content_items
UPDATE user_preferences

Service Calls#

text
{SERVICE_NAME}.{METHOD_NAME}

Examples:
auth.validateToken
content.getCourse
ai.generateResponse

Background Jobs#

text
job.{JOB_TYPE}

Examples:
job.processMediaUpload
job.sendNotification
job.syncUserData

Context Propagation#

HTTP Headers#

typescript
// Inject context into outgoing request
const headers = {};
injectTraceContext(currentSpan.context, headers);
// Result: { 'traceparent': '00-trace-span-01' }

// Extract context from incoming request
const context = extractTraceContext(request.headers);
tracer.startSpanWithContext('operation', context, async (span) => {
  // Child span linked to parent trace
});

Kafka Messages#

typescript
// Producer
const message = {
  key: userId,
  value: JSON.stringify(event),
  headers: {
    traceparent: `00-${span.context.traceId}-${span.context.spanId}-01`,
  },
};

// Consumer
const traceparent = message.headers.traceparent?.toString();
const context = parseTraceparent(traceparent);
tracer.startSpanWithContext('process-message', context, async (span) => {
  // Process message
});

gRPC Metadata#

typescript
// Client
const metadata = new grpc.Metadata();
metadata.set('traceparent', formatTraceparent(span.context));

// Server
const traceparent = call.metadata.get('traceparent')[0];

Sampling Strategies#

Head-Based Sampling#

Sample decision made at trace start:

typescript
const tracer = createTracer({
  serviceName: 'my-service',
  sampleRate: 0.1, // Sample 10% of traces
});

Adaptive Sampling#

typescript
const tracer = createTracer({
  serviceName: 'my-service',
  sampleRate: calculateSampleRate(),
});

function calculateSampleRate(): number {
  const currentRPS = getRequestsPerSecond();

  if (currentRPS > 10000) return 0.01; // 1% at high load
  if (currentRPS > 1000) return 0.1; // 10% at medium load
  return 1.0; // 100% at low load
}

Error-Based Sampling#

Always sample errors:

typescript
app.addHook('onError', async (request, reply, error) => {
  // Force sample on error
  request.span.context.traceFlags = 1;

  request.span.setStatus({
    code: SpanStatusCode.ERROR,
    message: error.message,
  });
});

High-Value Request Sampling#

typescript
// Always sample specific operations
if (request.url.includes('/payment') || request.url.includes('/checkout')) {
  span.context.traceFlags = 1; // Force sample
}

Correlation with Logs and Metrics#

Log Correlation#

typescript
import { logger } from '@lilith/logger';

app.addHook('onRequest', (request, reply) => {
  // Add trace context to logger
  request.log = logger.child({
    traceId: request.span?.context.traceId,
    spanId: request.span?.context.spanId,
  });
});

// Usage in handlers
request.log.info('Processing request');
// Output: {"message":"Processing request","traceId":"abc123","spanId":"def456"}

Metric Correlation#

typescript
import { histogram } from '@lilith/metrics';

const requestDuration = histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'status', 'trace_sampled'],
});

app.addHook('onResponse', (request, reply) => {
  requestDuration.observe(
    {
      method: request.method,
      status: reply.statusCode,
      trace_sampled: request.span?.context.traceFlags === 1 ? 'true' : 'false',
    },
    reply.getResponseTime() / 1000
  );
});

Exemplars (Linking Metrics to Traces)#

typescript
// Add trace ID as exemplar to histogram
requestDuration.observe({ method: 'GET', status: '200' }, duration, {
  traceId: span.context.traceId,
});

Troubleshooting#

Missing Traces#

  1. Check sampling rate

    typescript
    // Temporarily set to 100%
    sampleRate: 1.0;
    
  2. Verify context propagation

    bash
    # Check incoming headers
    curl -v -H "traceparent: 00-abc123-def456-01" http://localhost:3000/api/test
    
  3. Check collector connectivity

    bash
    kubectl logs -n monitoring deploy/otel-collector
    

Broken Traces#

  1. Context not propagated

    typescript
    // Ensure headers are forwarded
    const headers = {};
    injectTraceContext(span.context, headers);
    
  2. Async context lost

    typescript
    // Use context management
    const activeSpan = tracer.getActiveSpan();
    

Performance Issues#

  1. Too many attributes

    typescript
    // Limit attribute count
    const MAX_ATTRIBUTES = 32;
    
  2. Large payloads

    typescript
    // Don't trace large request/response bodies
    span.setAttribute('http.request_content_length', body.length);
    // NOT: span.setAttribute('http.request_body', body);
    

Debug Commands#

bash
# Check trace in Tempo
curl -G http://tempo:3200/api/traces/{traceId}

# Query Tempo via Grafana
# TraceQL: {service.name="auth-service" && status=error}

# View recent traces
kubectl port-forward -n monitoring svc/tempo 3200:3200
curl http://localhost:3200/api/traces?service=auth-service&limit=10