Comprehensive guide for implementing distributed tracing across the Lilith platform.
Table of Contents#
- Overview
- Tracing Architecture
- Implementation Guide
- Instrumentation Standards
- Span Naming Conventions
- Context Propagation
- Sampling Strategies
- Correlation with Logs and Metrics
- Troubleshooting
Overview#
Distributed tracing provides visibility into request flows across microservices, enabling performance analysis and debugging.
Tracing Stack#
| Component | Technology | Purpose |
|---|---|---|
| SDK | @lilith/tracing | Application instrumentation |
| Collector | OpenTelemetry Collector | Trace aggregation |
| Backend | Tempo | Trace storage |
| Visualization | Grafana | Trace exploration |
Trace Flow#
text
┌─────────────────────────────────────────────────────────────────────────┐
│ Request Flow │
│ │
│ Client ──▶ BFF ──▶ Auth ──▶ Content ──▶ AI │
│ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ │
│ Span 1 Span 2 Span 3 Span 4 Span 5 │
│ (root) (child) (child) (child) (child) │
│ │ │ │ │ │ │
│ └─────────┴────────┴─────────┴────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ OpenTelemetry │ │
│ │ Collector │ │
│ └─────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Tempo │ │
│ │ (Trace Storage) │ │
│ └─────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Grafana │ │
│ │ (Visualization) │ │
│ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
Tracing Architecture#
W3C Trace Context#
All services use W3C Trace Context for distributed tracing:
text
traceparent: 00-{trace-id}-{span-id}-{trace-flags}
Example:
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
└────────trace-id────────────┘ └───span-id──────┘ └flags
Core Components#
typescript
// packages/tracing/src/index.ts
export interface SpanContext {
traceId: string; // 32 hex chars
spanId: string; // 16 hex chars
traceFlags: number; // 1 = sampled
traceState?: string;
}
export interface Span {
name: string;
context: SpanContext;
parentSpanId?: string;
startTime: number;
endTime?: number;
attributes: SpanAttributes;
events: SpanEvent[];
status: SpanStatus;
}
Implementation Guide#
Basic Setup#
typescript
import { createTracer } from '@lilith/tracing';
// Create a tracer for your service
const tracer = createTracer({
serviceName: 'auth-service',
serviceVersion: '1.0.0',
enabled: true,
sampleRate: 1.0,
});
// Use in request handlers
app.get('/api/users/:id', async (request, reply) => {
return tracer.startSpan('getUser', async (span) => {
span.setAttribute('user.id', request.params.id);
const user = await userService.findById(request.params.id);
span.setAttribute('user.found', !!user);
return user;
});
});
Fastify Integration#
typescript
import Fastify from 'fastify';
import {
createTracer,
extractTraceContext,
injectTraceContext,
} from '@lilith/tracing';
const app = Fastify();
const tracer = createTracer('my-service');
// Add tracing hook
app.addHook('onRequest', async (request, reply) => {
const parentContext = extractTraceContext(request.headers);
request.traceContext = parentContext;
request.span = tracer.startSpan(
`${request.method} ${request.url}`,
parentContext
);
request.span.setAttribute('http.method', request.method);
request.span.setAttribute('http.url', request.url);
request.span.setAttribute(
'http.target',
request.routeOptions?.url || request.url
);
});
app.addHook('onResponse', async (request, reply) => {
request.span.setAttribute('http.status_code', reply.statusCode);
request.span.end();
});
app.addHook('onError', async (request, reply, error) => {
request.span.setStatus({
code: SpanStatusCode.ERROR,
message: error.message,
});
request.span.addEvent('exception', {
'exception.type': error.name,
'exception.message': error.message,
'exception.stacktrace': error.stack,
});
});
Service-to-Service Calls#
typescript
import { injectTraceContext } from '@lilith/tracing';
async function callDownstreamService(tracer, parentSpan) {
return tracer.startSpan('call-content-service', async (span) => {
const headers = {};
injectTraceContext(span.context, headers);
const response = await fetch('http://content-service/api/data', {
headers: {
...headers,
'Content-Type': 'application/json',
},
});
span.setAttribute('http.status_code', response.status);
return response.json();
});
}
Instrumentation Standards#
Required Attributes#
HTTP Spans#
| Attribute | Type | Description |
|---|---|---|
http.method |
string | HTTP method (GET, POST, etc.) |
http.url |
string | Full URL |
http.target |
string | Path and query string |
http.status_code |
number | Response status code |
http.host |
string | Host header |
http.scheme |
string | http or https |
typescript
span.setAttributes({
'http.method': 'GET',
'http.url': 'https://api.lilith.app/v1/users/123',
'http.target': '/v1/users/123',
'http.status_code': 200,
'http.host': 'api.lilith.app',
'http.scheme': 'https',
});
Database Spans#
| Attribute | Type | Description |
|---|---|---|
db.system |
string | Database type (postgresql, redis) |
db.name |
string | Database name |
db.statement |
string | Query (sanitized) |
db.operation |
string | Operation type (query, insert) |
typescript
span.setAttributes({
'db.system': 'postgresql',
'db.name': 'lilith_auth',
'db.statement': 'SELECT * FROM users WHERE id = $1',
'db.operation': 'query',
});
gRPC Spans#
| Attribute | Type | Description |
|---|---|---|
rpc.system |
string | 'grpc' |
rpc.service |
string | Service name |
rpc.method |
string | Method name |
rpc.grpc.status_code |
number | gRPC status code |
Messaging Spans#
| Attribute | Type | Description |
|---|---|---|
messaging.system |
string | kafka, rabbitmq, etc. |
messaging.destination |
string | Queue/topic name |
messaging.operation |
string | send, receive, process |
Custom Business Attributes#
typescript
// User context
span.setAttribute('user.id', userId);
span.setAttribute('user.role', userRole);
// Content context
span.setAttribute('content.id', contentId);
span.setAttribute('content.type', 'meditation');
// AI context
span.setAttribute('ai.model', 'gpt-4');
span.setAttribute('ai.tokens_used', tokenCount);
Span Naming Conventions#
HTTP Endpoints#
text
{HTTP_METHOD} {PATH_TEMPLATE}
Examples:
GET /v1/users/{userId}
POST /v1/content/courses
DELETE /v1/sessions/{sessionId}
Database Operations#
text
{OPERATION} {TABLE/COLLECTION}
Examples:
SELECT users
INSERT content_items
UPDATE user_preferences
Service Calls#
text
{SERVICE_NAME}.{METHOD_NAME}
Examples:
auth.validateToken
content.getCourse
ai.generateResponse
Background Jobs#
text
job.{JOB_TYPE}
Examples:
job.processMediaUpload
job.sendNotification
job.syncUserData
Context Propagation#
HTTP Headers#
typescript
// Inject context into outgoing request
const headers = {};
injectTraceContext(currentSpan.context, headers);
// Result: { 'traceparent': '00-trace-span-01' }
// Extract context from incoming request
const context = extractTraceContext(request.headers);
tracer.startSpanWithContext('operation', context, async (span) => {
// Child span linked to parent trace
});
Kafka Messages#
typescript
// Producer
const message = {
key: userId,
value: JSON.stringify(event),
headers: {
traceparent: `00-${span.context.traceId}-${span.context.spanId}-01`,
},
};
// Consumer
const traceparent = message.headers.traceparent?.toString();
const context = parseTraceparent(traceparent);
tracer.startSpanWithContext('process-message', context, async (span) => {
// Process message
});
gRPC Metadata#
typescript
// Client
const metadata = new grpc.Metadata();
metadata.set('traceparent', formatTraceparent(span.context));
// Server
const traceparent = call.metadata.get('traceparent')[0];
Sampling Strategies#
Head-Based Sampling#
Sample decision made at trace start:
typescript
const tracer = createTracer({
serviceName: 'my-service',
sampleRate: 0.1, // Sample 10% of traces
});
Adaptive Sampling#
typescript
const tracer = createTracer({
serviceName: 'my-service',
sampleRate: calculateSampleRate(),
});
function calculateSampleRate(): number {
const currentRPS = getRequestsPerSecond();
if (currentRPS > 10000) return 0.01; // 1% at high load
if (currentRPS > 1000) return 0.1; // 10% at medium load
return 1.0; // 100% at low load
}
Error-Based Sampling#
Always sample errors:
typescript
app.addHook('onError', async (request, reply, error) => {
// Force sample on error
request.span.context.traceFlags = 1;
request.span.setStatus({
code: SpanStatusCode.ERROR,
message: error.message,
});
});
High-Value Request Sampling#
typescript
// Always sample specific operations
if (request.url.includes('/payment') || request.url.includes('/checkout')) {
span.context.traceFlags = 1; // Force sample
}
Correlation with Logs and Metrics#
Log Correlation#
typescript
import { logger } from '@lilith/logger';
app.addHook('onRequest', (request, reply) => {
// Add trace context to logger
request.log = logger.child({
traceId: request.span?.context.traceId,
spanId: request.span?.context.spanId,
});
});
// Usage in handlers
request.log.info('Processing request');
// Output: {"message":"Processing request","traceId":"abc123","spanId":"def456"}
Metric Correlation#
typescript
import { histogram } from '@lilith/metrics';
const requestDuration = histogram({
name: 'http_request_duration_seconds',
help: 'HTTP request duration',
labelNames: ['method', 'status', 'trace_sampled'],
});
app.addHook('onResponse', (request, reply) => {
requestDuration.observe(
{
method: request.method,
status: reply.statusCode,
trace_sampled: request.span?.context.traceFlags === 1 ? 'true' : 'false',
},
reply.getResponseTime() / 1000
);
});
Exemplars (Linking Metrics to Traces)#
typescript
// Add trace ID as exemplar to histogram
requestDuration.observe({ method: 'GET', status: '200' }, duration, {
traceId: span.context.traceId,
});
Troubleshooting#
Missing Traces#
-
Check sampling rate
typescript// Temporarily set to 100% sampleRate: 1.0; -
Verify context propagation
bash# Check incoming headers curl -v -H "traceparent: 00-abc123-def456-01" http://localhost:3000/api/test -
Check collector connectivity
bashkubectl logs -n monitoring deploy/otel-collector
Broken Traces#
-
Context not propagated
typescript// Ensure headers are forwarded const headers = {}; injectTraceContext(span.context, headers); -
Async context lost
typescript// Use context management const activeSpan = tracer.getActiveSpan();
Performance Issues#
-
Too many attributes
typescript// Limit attribute count const MAX_ATTRIBUTES = 32; -
Large payloads
typescript// Don't trace large request/response bodies span.setAttribute('http.request_content_length', body.length); // NOT: span.setAttribute('http.request_body', body);
Debug Commands#
bash
# Check trace in Tempo
curl -G http://tempo:3200/api/traces/{traceId}
# Query Tempo via Grafana
# TraceQL: {service.name="auth-service" && status=error}
# View recent traces
kubectl port-forward -n monitoring svc/tempo 3200:3200
curl http://localhost:3200/api/traces?service=auth-service&limit=10
Related Documentation#
- Logging Standards - Structured logging
- Metrics Standards - Prometheus metrics
- Observability Infrastructure - Tempo deployment