Document Version: 1.0 Configuration Date: 2026-02-01 Platform: Psyche AI Virtual Assistant Status: OPERATIONAL - Monitoring dashboard ready
Executive Summary#
This document provides the comprehensive monitoring configuration for the Psyche
AI Virtual Assistant Platform launch. The monitoring stack leverages the
existing @oshun/metrics, @oshun/tracing, and domain-specific metrics
collectors to provide real-time visibility into avatar rendering, voice
pipeline, error rates, and GPU utilization.
Monitoring Stack Overview#
| Component | Technology | Purpose |
|---|---|---|
| Metrics Collection | prom-client | Prometheus-compatible metrics |
| Distributed Tracing | OpenTelemetry | Request tracing |
| Metrics Storage | Prometheus | Time-series database |
| Visualization | Grafana | Dashboards and alerts |
| Alerting | AlertManager | Alert routing |
1. Metrics Architecture#
1.1 Metrics Endpoint Configuration#
yaml
# Service Metrics Exposure
services:
avatar-engine:
metrics_port: 9090
metrics_path: /metrics
health_path: /health
ready_path: /ready
voice-engine:
metrics_port: 9091
metrics_path: /metrics
health_path: /health
ready_path: /ready
session-manager:
metrics_port: 9092
metrics_path: /metrics
health_path: /health
ready_path: /ready
memory-service:
metrics_port: 9093
metrics_path: /metrics
health_path: /health
ready_path: /ready
1.2 Prometheus Scrape Configuration#
yaml
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
# Psyche Avatar Engine
- job_name: 'psyche-avatar-engine'
static_configs:
- targets: ['avatar-engine:9090']
relabel_configs:
- source_labels: [__address__]
target_label: service
replacement: 'avatar-engine'
# Psyche Voice Engine
- job_name: 'psyche-voice-engine'
static_configs:
- targets: ['voice-engine:9091']
relabel_configs:
- source_labels: [__address__]
target_label: service
replacement: 'voice-engine'
# Psyche Session Manager
- job_name: 'psyche-session-manager'
static_configs:
- targets: ['session-manager:9092']
relabel_configs:
- source_labels: [__address__]
target_label: service
replacement: 'session-manager'
# Psyche Memory Service
- job_name: 'psyche-memory-service'
static_configs:
- targets: ['memory-service:9093']
relabel_configs:
- source_labels: [__address__]
target_label: service
replacement: 'memory-service'
# GPU Endpoints (RunPod/Cloud)
- job_name: 'psyche-gpu-endpoints'
kubernetes_sd_configs:
- role: endpoints
namespaces:
names: ['psyche-gpu']
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
action: keep
regex: psyche-gpu-.*
2. Avatar Rendering Metrics#
2.1 Core Metrics#
typescript
// Avatar Engine Metrics
const avatarMetrics = {
// Frame Performance
avatar_frame_render_duration_seconds: Histogram({
name: 'avatar_frame_render_duration_seconds',
help: 'Time to render a single avatar frame',
labelNames: ['resolution', 'quality', 'gpu_type'],
buckets: [0.005, 0.01, 0.016, 0.025, 0.033, 0.05, 0.1, 0.25],
}),
avatar_frames_per_second: Gauge({
name: 'avatar_frames_per_second',
help: 'Current avatar rendering FPS',
labelNames: ['session_id', 'quality'],
}),
// 3D Gaussian Splatting
avatar_3dgs_points_total: Gauge({
name: 'avatar_3dgs_points_total',
help: 'Number of 3DGS points in current model',
labelNames: ['avatar_id'],
}),
avatar_3dgs_render_latency_seconds: Histogram({
name: 'avatar_3dgs_render_latency_seconds',
help: '3DGS rendering latency',
labelNames: ['resolution'],
buckets: [0.005, 0.01, 0.02, 0.033, 0.05, 0.1],
}),
// FLAME Expression
avatar_flame_inference_duration_seconds: Histogram({
name: 'avatar_flame_inference_duration_seconds',
help: 'FLAME model inference time',
labelNames: ['expression_type'],
buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1],
}),
avatar_expression_coefficients_updated_total: Counter({
name: 'avatar_expression_coefficients_updated_total',
help: 'Number of expression coefficient updates',
labelNames: ['expression_type'],
}),
// Quality Metrics
avatar_ssim_score: Gauge({
name: 'avatar_ssim_score',
help: 'Structural similarity index (0-1)',
labelNames: ['avatar_id'],
}),
avatar_psnr_score: Gauge({
name: 'avatar_psnr_score',
help: 'Peak signal-to-noise ratio (dB)',
labelNames: ['avatar_id'],
}),
};
2.2 Rendering Performance Targets#
| Metric | Target | Warning | Critical |
|---|---|---|---|
| Frame Rate | >= 30 FPS | < 25 FPS | < 15 FPS |
| Frame Render Time | < 33ms | > 40ms | > 66ms |
| 3DGS Render Latency | < 20ms | > 30ms | > 50ms |
| FLAME Inference | < 10ms | > 20ms | > 50ms |
| SSIM Score | > 0.92 | < 0.90 | < 0.85 |
2.3 Dashboard Panel: Avatar Rendering#
json
{
"title": "Avatar Rendering Performance",
"type": "row",
"panels": [
{
"title": "Frame Rate (FPS)",
"type": "stat",
"targets": [
{
"expr": "avg(avatar_frames_per_second)",
"legendFormat": "Avg FPS"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": 0 },
{ "color": "yellow", "value": 25 },
{ "color": "green", "value": 30 }
]
}
},
{
"title": "Render Latency P95",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(avatar_frame_render_duration_seconds_bucket[5m]))",
"legendFormat": "P95"
}
],
"unit": "s"
},
{
"title": "Frame Rate Over Time",
"type": "timeseries",
"targets": [
{
"expr": "avatar_frames_per_second",
"legendFormat": "{{session_id}}"
}
]
},
{
"title": "Quality Score (SSIM)",
"type": "gauge",
"targets": [
{
"expr": "avg(avatar_ssim_score)",
"legendFormat": "SSIM"
}
],
"min": 0,
"max": 1
}
]
}
3. Voice Pipeline Metrics#
3.1 Core Metrics#
typescript
// Voice Engine Metrics
const voiceMetrics = {
// End-to-End Latency
voice_e2e_latency_seconds: Histogram({
name: 'voice_e2e_latency_seconds',
help: 'End-to-end voice pipeline latency',
labelNames: ['session_id'],
buckets: [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 5],
}),
// Speech-to-Text
voice_stt_latency_seconds: Histogram({
name: 'voice_stt_latency_seconds',
help: 'Speech-to-text processing latency',
labelNames: ['provider', 'language'],
buckets: [0.05, 0.1, 0.15, 0.2, 0.3, 0.5, 1],
}),
voice_stt_requests_total: Counter({
name: 'voice_stt_requests_total',
help: 'Total STT requests',
labelNames: ['provider', 'status'],
}),
voice_stt_word_error_rate: Gauge({
name: 'voice_stt_word_error_rate',
help: 'Word error rate (0-1)',
labelNames: ['provider'],
}),
// LLM Processing
voice_llm_latency_seconds: Histogram({
name: 'voice_llm_latency_seconds',
help: 'LLM inference latency',
labelNames: ['model', 'persona_id'],
buckets: [0.1, 0.25, 0.5, 1, 2, 3, 5, 10],
}),
voice_llm_tokens_total: Counter({
name: 'voice_llm_tokens_total',
help: 'Total LLM tokens processed',
labelNames: ['model', 'type'], // type: input, output
}),
// Text-to-Speech
voice_tts_latency_seconds: Histogram({
name: 'voice_tts_latency_seconds',
help: 'Text-to-speech synthesis latency',
labelNames: ['provider', 'voice_id'],
buckets: [0.025, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5],
}),
voice_tts_requests_total: Counter({
name: 'voice_tts_requests_total',
help: 'Total TTS requests',
labelNames: ['provider', 'status'],
}),
voice_tts_characters_total: Counter({
name: 'voice_tts_characters_total',
help: 'Total characters synthesized',
labelNames: ['provider'],
}),
// Audio Quality
voice_audio_quality_score: Gauge({
name: 'voice_audio_quality_score',
help: 'Audio quality score (0-1)',
labelNames: ['session_id'],
}),
voice_lip_sync_accuracy: Gauge({
name: 'voice_lip_sync_accuracy',
help: 'Lip sync accuracy score (0-1)',
labelNames: ['session_id'],
}),
};
3.2 Voice Latency Targets#
| Component | Target | Warning | Critical |
|---|---|---|---|
| Speech-to-Text | < 100ms | > 150ms | > 300ms |
| LLM Inference | < 200ms | > 500ms | > 1000ms |
| Text-to-Speech | < 100ms | > 150ms | > 300ms |
| End-to-End Total | < 500ms | > 750ms | > 1500ms |
3.3 Dashboard Panel: Voice Pipeline#
json
{
"title": "Voice Pipeline Performance",
"type": "row",
"panels": [
{
"title": "E2E Latency P95",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m]))",
"legendFormat": "P95"
}
],
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "yellow", "value": 0.5 },
{ "color": "red", "value": 1 }
]
}
},
{
"title": "Latency Breakdown",
"type": "bargauge",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(voice_stt_latency_seconds_bucket[5m]))",
"legendFormat": "STT"
},
{
"expr": "histogram_quantile(0.95, rate(voice_llm_latency_seconds_bucket[5m]))",
"legendFormat": "LLM"
},
{
"expr": "histogram_quantile(0.95, rate(voice_tts_latency_seconds_bucket[5m]))",
"legendFormat": "TTS"
}
],
"unit": "s"
},
{
"title": "E2E Latency Distribution",
"type": "heatmap",
"targets": [
{
"expr": "rate(voice_e2e_latency_seconds_bucket[5m])",
"format": "heatmap"
}
]
},
{
"title": "Audio Quality Score",
"type": "gauge",
"targets": [
{
"expr": "avg(voice_audio_quality_score)",
"legendFormat": "Quality"
}
],
"min": 0,
"max": 1
}
]
}
4. Error Rate Metrics#
4.1 Core Metrics#
typescript
// Error Tracking Metrics
const errorMetrics = {
// HTTP Errors
http_requests_total: Counter({
name: 'http_requests_total',
help: 'Total HTTP requests',
labelNames: ['service', 'method', 'path', 'status'],
}),
http_errors_total: Counter({
name: 'http_errors_total',
help: 'Total HTTP errors (4xx, 5xx)',
labelNames: ['service', 'method', 'path', 'status', 'error_type'],
}),
// Service Errors
service_errors_total: Counter({
name: 'service_errors_total',
help: 'Total service-level errors',
labelNames: ['service', 'error_code', 'severity'],
}),
// Platform-Specific Errors
avatar_errors_total: Counter({
name: 'avatar_errors_total',
help: 'Avatar rendering errors',
labelNames: ['error_type', 'gpu_id'],
}),
voice_errors_total: Counter({
name: 'voice_errors_total',
help: 'Voice pipeline errors',
labelNames: ['component', 'error_type', 'provider'],
}),
conferencing_errors_total: Counter({
name: 'conferencing_errors_total',
help: 'Video conferencing errors',
labelNames: ['platform', 'error_type'],
}),
// Error Rate (computed)
error_rate: Gauge({
name: 'error_rate',
help: 'Current error rate (0-1)',
labelNames: ['service'],
}),
};
4.2 Error Rate Targets#
| Service | Target | Warning | Critical |
|---|---|---|---|
| Avatar Engine | < 0.1% | > 0.5% | > 1% |
| Voice Engine | < 0.5% | > 1% | > 2% |
| Session Manager | < 0.1% | > 0.5% | > 1% |
| Memory Service | < 0.1% | > 0.5% | > 1% |
| Conferencing (Zoom/Teams) | < 1% | > 2% | > 5% |
4.3 Dashboard Panel: Error Rates#
json
{
"title": "Error Rate Monitoring",
"type": "row",
"panels": [
{
"title": "Overall Error Rate",
"type": "stat",
"targets": [
{
"expr": "sum(rate(http_errors_total[5m])) / sum(rate(http_requests_total[5m])) * 100",
"legendFormat": "Error %"
}
],
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "yellow", "value": 0.5 },
{ "color": "red", "value": 1 }
]
}
},
{
"title": "Error Rate by Service",
"type": "timeseries",
"targets": [
{
"expr": "sum by(service) (rate(http_errors_total[5m])) / sum by(service) (rate(http_requests_total[5m])) * 100",
"legendFormat": "{{service}}"
}
],
"unit": "percent"
},
{
"title": "Error Types Distribution",
"type": "piechart",
"targets": [
{
"expr": "sum by(error_type) (increase(service_errors_total[1h]))",
"legendFormat": "{{error_type}}"
}
]
},
{
"title": "Recent Errors",
"type": "table",
"targets": [
{
"expr": "topk(10, sum by(service, error_code) (increase(service_errors_total[5m])))",
"format": "table"
}
]
}
]
}
5. GPU Utilization Metrics#
5.1 Core Metrics#
typescript
// GPU Metrics (from @oshun/gpu-dispatcher)
const gpuMetrics = {
// Utilization
gpu_utilization_percent: Gauge({
name: 'gpu_utilization_percent',
help: 'GPU compute utilization percentage',
labelNames: ['gpu_id', 'gpu_type'],
}),
gpu_memory_used_bytes: Gauge({
name: 'gpu_memory_used_bytes',
help: 'GPU memory used in bytes',
labelNames: ['gpu_id', 'gpu_type'],
}),
gpu_memory_total_bytes: Gauge({
name: 'gpu_memory_total_bytes',
help: 'Total GPU memory in bytes',
labelNames: ['gpu_id', 'gpu_type'],
}),
gpu_temperature_celsius: Gauge({
name: 'gpu_temperature_celsius',
help: 'GPU temperature in Celsius',
labelNames: ['gpu_id'],
}),
gpu_power_watts: Gauge({
name: 'gpu_power_watts',
help: 'GPU power consumption in watts',
labelNames: ['gpu_id'],
}),
// Job Processing
gpu_jobs_total: Counter({
name: 'gpu_jobs_total',
help: 'Total GPU jobs submitted',
labelNames: ['job_type', 'priority'],
}),
gpu_jobs_completed_total: Counter({
name: 'gpu_jobs_completed_total',
help: 'Completed GPU jobs',
labelNames: ['job_type', 'status'],
}),
gpu_jobs_pending: Gauge({
name: 'gpu_jobs_pending',
help: 'Jobs waiting in queue',
labelNames: ['job_type'],
}),
gpu_jobs_running: Gauge({
name: 'gpu_jobs_running',
help: 'Currently running jobs',
labelNames: ['gpu_id'],
}),
gpu_job_duration_seconds: Histogram({
name: 'gpu_job_duration_seconds',
help: 'GPU job execution duration',
labelNames: ['job_type'],
buckets: [0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600],
}),
gpu_queue_wait_seconds: Histogram({
name: 'gpu_queue_wait_seconds',
help: 'Time spent waiting in queue',
labelNames: ['job_type', 'priority'],
buckets: [0.1, 0.5, 1, 2, 5, 10, 30, 60, 120],
}),
// Endpoint Health
gpu_endpoint_health: Gauge({
name: 'gpu_endpoint_health',
help: 'Endpoint health (0=unhealthy, 0.5=degraded, 1=healthy)',
labelNames: ['endpoint_id', 'gpu_type'],
}),
gpu_circuit_breaker_state: Gauge({
name: 'gpu_circuit_breaker_state',
help: 'Circuit breaker state (0=closed, 0.5=half-open, 1=open)',
labelNames: ['endpoint_id'],
}),
// Cost Tracking
gpu_cost_cents_total: Counter({
name: 'gpu_cost_cents_total',
help: 'Total GPU cost in cents',
labelNames: ['gpu_type', 'job_type'],
}),
gpu_seconds_total: Counter({
name: 'gpu_seconds_total',
help: 'Total GPU seconds consumed',
labelNames: ['gpu_type'],
}),
};
5.2 GPU Utilization Targets#
| Metric | Target | Warning | Critical |
|---|---|---|---|
| GPU Utilization | 60-80% | > 90% | > 95% |
| Memory Utilization | 60-80% | > 85% | > 95% |
| Temperature | < 75°C | > 80°C | > 85°C |
| Queue Depth | < 10 | > 50 | > 100 |
| Queue Wait Time (P95) | < 5s | > 30s | > 60s |
5.3 Dashboard Panel: GPU Utilization#
json
{
"title": "GPU Utilization",
"type": "row",
"panels": [
{
"title": "GPU Pool Status",
"type": "stat",
"targets": [
{
"expr": "count(gpu_endpoint_health == 1)",
"legendFormat": "Healthy"
},
{
"expr": "count(gpu_endpoint_health == 0.5)",
"legendFormat": "Degraded"
},
{
"expr": "count(gpu_endpoint_health == 0)",
"legendFormat": "Unhealthy"
}
]
},
{
"title": "GPU Utilization by Node",
"type": "timeseries",
"targets": [
{
"expr": "gpu_utilization_percent",
"legendFormat": "{{gpu_id}}"
}
],
"unit": "percent"
},
{
"title": "GPU Memory Usage",
"type": "gauge",
"targets": [
{
"expr": "sum(gpu_memory_used_bytes) / sum(gpu_memory_total_bytes) * 100",
"legendFormat": "Memory %"
}
],
"unit": "percent",
"max": 100
},
{
"title": "Job Queue Depth",
"type": "timeseries",
"targets": [
{
"expr": "sum(gpu_jobs_pending)",
"legendFormat": "Pending"
},
{
"expr": "sum(gpu_jobs_running)",
"legendFormat": "Running"
}
]
},
{
"title": "GPU Temperature",
"type": "timeseries",
"targets": [
{
"expr": "gpu_temperature_celsius",
"legendFormat": "{{gpu_id}}"
}
],
"unit": "celsius",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": 0 },
{ "color": "yellow", "value": 75 },
{ "color": "red", "value": 85 }
]
}
},
{
"title": "Circuit Breaker Status",
"type": "table",
"targets": [
{
"expr": "gpu_circuit_breaker_state",
"format": "table"
}
]
}
]
}
6. Alert Rules Configuration#
6.1 Avatar Rendering Alerts#
yaml
groups:
- name: psyche-avatar-alerts
rules:
- alert: AvatarLowFPS
expr: avatar_frames_per_second < 25
for: 2m
labels:
severity: warning
service: avatar-engine
annotations:
summary: 'Avatar rendering FPS below threshold'
description:
'FPS is {{ $value }} for session {{ $labels.session_id }}'
- alert: AvatarCriticalFPS
expr: avatar_frames_per_second < 15
for: 1m
labels:
severity: critical
service: avatar-engine
annotations:
summary: 'Avatar rendering FPS critically low'
description:
'FPS is {{ $value }} for session {{ $labels.session_id }}'
- alert: AvatarRenderLatencyHigh
expr:
histogram_quantile(0.95,
rate(avatar_frame_render_duration_seconds_bucket[5m])) > 0.04
for: 3m
labels:
severity: warning
service: avatar-engine
annotations:
summary: 'Avatar render latency P95 above 40ms'
description: 'P95 latency is {{ $value }}s'
- alert: AvatarQualityDegraded
expr: avg(avatar_ssim_score) < 0.90
for: 5m
labels:
severity: warning
service: avatar-engine
annotations:
summary: 'Avatar rendering quality degraded'
description: 'SSIM score is {{ $value }}'
6.2 Voice Pipeline Alerts#
yaml
groups:
- name: psyche-voice-alerts
rules:
- alert: VoiceE2ELatencyHigh
expr:
histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m])) >
0.75
for: 3m
labels:
severity: warning
service: voice-engine
annotations:
summary: 'Voice E2E latency P95 above 750ms'
description: 'P95 latency is {{ $value }}s'
- alert: VoiceE2ELatencyCritical
expr:
histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m])) >
1.5
for: 1m
labels:
severity: critical
service: voice-engine
annotations:
summary: 'Voice E2E latency P95 above 1.5s'
description: 'P95 latency is {{ $value }}s'
- alert: STTLatencyHigh
expr:
histogram_quantile(0.95, rate(voice_stt_latency_seconds_bucket[5m])) >
0.15
for: 3m
labels:
severity: warning
service: voice-engine
component: stt
annotations:
summary: 'STT latency P95 above 150ms'
description: 'P95 latency is {{ $value }}s'
- alert: TTSLatencyHigh
expr:
histogram_quantile(0.95, rate(voice_tts_latency_seconds_bucket[5m])) >
0.15
for: 3m
labels:
severity: warning
service: voice-engine
component: tts
annotations:
summary: 'TTS latency P95 above 150ms'
description: 'P95 latency is {{ $value }}s'
6.3 Error Rate Alerts#
yaml
groups:
- name: psyche-error-alerts
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_errors_total[5m]))
/ sum(rate(http_requests_total[5m])) > 0.01
for: 5m
labels:
severity: warning
annotations:
summary: 'Error rate above 1%'
description: 'Error rate is {{ $value | humanizePercentage }}'
- alert: CriticalErrorRate
expr: |
sum(rate(http_errors_total[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
for: 2m
labels:
severity: critical
annotations:
summary: 'Error rate above 5%'
description: 'Error rate is {{ $value | humanizePercentage }}'
- alert: AvatarErrorsIncreasing
expr: rate(avatar_errors_total[5m]) > 1
for: 3m
labels:
severity: warning
service: avatar-engine
annotations:
summary: 'Avatar errors increasing'
description: 'Error rate is {{ $value }}/s'
- alert: VoiceErrorsIncreasing
expr: rate(voice_errors_total[5m]) > 1
for: 3m
labels:
severity: warning
service: voice-engine
annotations:
summary: 'Voice pipeline errors increasing'
description: 'Error rate is {{ $value }}/s'
6.4 GPU Alerts#
yaml
groups:
- name: psyche-gpu-alerts
rules:
- alert: GPUHighUtilization
expr: gpu_utilization_percent > 90
for: 5m
labels:
severity: warning
annotations:
summary: 'GPU utilization above 90%'
description: 'GPU {{ $labels.gpu_id }} at {{ $value }}%'
- alert: GPUCriticalUtilization
expr: gpu_utilization_percent > 95
for: 2m
labels:
severity: critical
annotations:
summary: 'GPU utilization above 95%'
description: 'GPU {{ $labels.gpu_id }} at {{ $value }}%'
- alert: GPUMemoryExhaustion
expr: gpu_memory_used_bytes / gpu_memory_total_bytes > 0.95
for: 2m
labels:
severity: critical
annotations:
summary: 'GPU memory nearly exhausted'
description:
'GPU {{ $labels.gpu_id }} memory at {{ $value | humanizePercentage
}}'
- alert: GPUOverheating
expr: gpu_temperature_celsius > 85
for: 1m
labels:
severity: critical
annotations:
summary: 'GPU overheating'
description: 'GPU {{ $labels.gpu_id }} at {{ $value }}°C'
- alert: GPUQueueBacklog
expr: sum(gpu_jobs_pending) > 100
for: 5m
labels:
severity: warning
annotations:
summary: 'GPU job queue backlog'
description: '{{ $value }} jobs pending'
- alert: GPUCircuitBreakerOpen
expr: gpu_circuit_breaker_state == 1
for: 1m
labels:
severity: critical
annotations:
summary: 'GPU circuit breaker open'
description: 'Circuit breaker open for {{ $labels.endpoint_id }}'
- alert: GPUEndpointUnhealthy
expr: gpu_endpoint_health == 0
for: 2m
labels:
severity: critical
annotations:
summary: 'GPU endpoint unhealthy'
description: 'Endpoint {{ $labels.endpoint_id }} is unhealthy'
7. Grafana Dashboard Configuration#
7.1 Dashboard Manifest#
json
{
"dashboard": {
"title": "Psyche Platform - Launch Monitoring",
"uid": "psyche-launch-monitoring",
"tags": ["psyche", "launch", "production"],
"timezone": "browser",
"refresh": "10s",
"rows": [
{
"title": "Overview",
"panels": [
"service_health",
"active_sessions",
"error_rate",
"p95_latency"
]
},
{
"title": "Avatar Rendering",
"panels": ["fps", "render_latency", "quality_score", "gpu_memory"]
},
{
"title": "Voice Pipeline",
"panels": [
"e2e_latency",
"latency_breakdown",
"stt_metrics",
"tts_metrics"
]
},
{
"title": "Error Tracking",
"panels": [
"error_rate_by_service",
"error_distribution",
"recent_errors"
]
},
{
"title": "GPU Utilization",
"panels": [
"gpu_pool_status",
"gpu_utilization",
"job_queue",
"circuit_breakers"
]
},
{
"title": "Infrastructure",
"panels": ["database_pool", "cache_hit_rate", "rate_limiter_status"]
}
]
}
}
7.2 Key Dashboard Variables#
json
{
"templating": {
"list": [
{
"name": "service",
"type": "query",
"query": "label_values(up, service)",
"multi": true
},
{
"name": "gpu_id",
"type": "query",
"query": "label_values(gpu_utilization_percent, gpu_id)",
"multi": true
},
{
"name": "session_id",
"type": "query",
"query": "label_values(avatar_frames_per_second, session_id)",
"multi": false
}
]
}
}
8. AlertManager Configuration#
8.1 Routing Configuration#
yaml
# alertmanager.yml
global:
resolve_timeout: 5m
slack_api_url: '${SLACK_WEBHOOK_URL}'
pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
route:
group_by: ['alertname', 'service', 'severity']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'psyche-default'
routes:
# Critical alerts -> PagerDuty + Slack
- match:
severity: critical
receiver: 'psyche-critical'
continue: true
# Warning alerts -> Slack only
- match:
severity: warning
receiver: 'psyche-warnings'
# GPU-specific alerts
- match:
service: gpu
receiver: 'psyche-gpu-team'
receivers:
- name: 'psyche-default'
slack_configs:
- channel: '#psyche-alerts'
send_resolved: true
- name: 'psyche-critical'
pagerduty_configs:
- service_key: '${PAGERDUTY_SERVICE_KEY}'
severity: critical
slack_configs:
- channel: '#psyche-critical'
send_resolved: true
- name: 'psyche-warnings'
slack_configs:
- channel: '#psyche-warnings'
send_resolved: true
- name: 'psyche-gpu-team'
slack_configs:
- channel: '#psyche-gpu-ops'
send_resolved: true
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'service']
9. Launch Readiness Checklist#
9.1 Monitoring Components#
| Component | Status | Notes |
|---|---|---|
| Prometheus Deployed | READY | Scraping all services |
| Grafana Dashboards | READY | 6 dashboard panels configured |
| Alert Rules | READY | 15+ alert rules configured |
| AlertManager | READY | PagerDuty + Slack integration |
| GPU Metrics Collector | READY | All GPU nodes reporting |
| Voice Pipeline Metrics | READY | E2E latency tracking active |
| Avatar Rendering Metrics | READY | FPS and quality tracking active |
| Error Rate Tracking | READY | Per-service error rates |
9.2 SLO Targets#
| SLO | Target | Measurement |
|---|---|---|
| Availability | 99.9% | Uptime over 30 days |
| Avatar FPS | >= 30 FPS | P95 over 1 hour |
| Voice E2E Latency | < 500ms | P95 over 1 hour |
| Error Rate | < 0.1% | Rolling 1 hour average |
| GPU Job Queue Time | < 5s | P95 over 1 hour |
10. Sign-off#
| Role | Name | Date | Signature |
|---|---|---|---|
| SRE Lead | _ | ____ | __ |
| Platform Lead | _ | ____ | __ |
| Security Lead | _ | ____ | __ |
| VP Engineering | _ | ____ | __ |
Configuration generated: 2026-02-01 Platform: Psyche AI Virtual Assistant Document classification: Internal - Operations