Eve library

Psyche Platform - Launch Monitoring Configuration

Guide

Revision 1

On this page

Document Version: 1.0 Configuration Date: 2026-02-01 Platform: Psyche AI Virtual Assistant Status: OPERATIONAL - Monitoring dashboard ready


Executive Summary#

This document provides the comprehensive monitoring configuration for the Psyche AI Virtual Assistant Platform launch. The monitoring stack leverages the existing @oshun/metrics, @oshun/tracing, and domain-specific metrics collectors to provide real-time visibility into avatar rendering, voice pipeline, error rates, and GPU utilization.

Monitoring Stack Overview#

Component Technology Purpose
Metrics Collection prom-client Prometheus-compatible metrics
Distributed Tracing OpenTelemetry Request tracing
Metrics Storage Prometheus Time-series database
Visualization Grafana Dashboards and alerts
Alerting AlertManager Alert routing

1. Metrics Architecture#

1.1 Metrics Endpoint Configuration#

yaml
# Service Metrics Exposure
services:
  avatar-engine:
    metrics_port: 9090
    metrics_path: /metrics
    health_path: /health
    ready_path: /ready

  voice-engine:
    metrics_port: 9091
    metrics_path: /metrics
    health_path: /health
    ready_path: /ready

  session-manager:
    metrics_port: 9092
    metrics_path: /metrics
    health_path: /health
    ready_path: /ready

  memory-service:
    metrics_port: 9093
    metrics_path: /metrics
    health_path: /health
    ready_path: /ready

1.2 Prometheus Scrape Configuration#

yaml
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Psyche Avatar Engine
  - job_name: 'psyche-avatar-engine'
    static_configs:
      - targets: ['avatar-engine:9090']
    relabel_configs:
      - source_labels: [__address__]
        target_label: service
        replacement: 'avatar-engine'

  # Psyche Voice Engine
  - job_name: 'psyche-voice-engine'
    static_configs:
      - targets: ['voice-engine:9091']
    relabel_configs:
      - source_labels: [__address__]
        target_label: service
        replacement: 'voice-engine'

  # Psyche Session Manager
  - job_name: 'psyche-session-manager'
    static_configs:
      - targets: ['session-manager:9092']
    relabel_configs:
      - source_labels: [__address__]
        target_label: service
        replacement: 'session-manager'

  # Psyche Memory Service
  - job_name: 'psyche-memory-service'
    static_configs:
      - targets: ['memory-service:9093']
    relabel_configs:
      - source_labels: [__address__]
        target_label: service
        replacement: 'memory-service'

  # GPU Endpoints (RunPod/Cloud)
  - job_name: 'psyche-gpu-endpoints'
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names: ['psyche-gpu']
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_app]
        action: keep
        regex: psyche-gpu-.*

2. Avatar Rendering Metrics#

2.1 Core Metrics#

typescript
// Avatar Engine Metrics
const avatarMetrics = {
  // Frame Performance
  avatar_frame_render_duration_seconds: Histogram({
    name: 'avatar_frame_render_duration_seconds',
    help: 'Time to render a single avatar frame',
    labelNames: ['resolution', 'quality', 'gpu_type'],
    buckets: [0.005, 0.01, 0.016, 0.025, 0.033, 0.05, 0.1, 0.25],
  }),

  avatar_frames_per_second: Gauge({
    name: 'avatar_frames_per_second',
    help: 'Current avatar rendering FPS',
    labelNames: ['session_id', 'quality'],
  }),

  // 3D Gaussian Splatting
  avatar_3dgs_points_total: Gauge({
    name: 'avatar_3dgs_points_total',
    help: 'Number of 3DGS points in current model',
    labelNames: ['avatar_id'],
  }),

  avatar_3dgs_render_latency_seconds: Histogram({
    name: 'avatar_3dgs_render_latency_seconds',
    help: '3DGS rendering latency',
    labelNames: ['resolution'],
    buckets: [0.005, 0.01, 0.02, 0.033, 0.05, 0.1],
  }),

  // FLAME Expression
  avatar_flame_inference_duration_seconds: Histogram({
    name: 'avatar_flame_inference_duration_seconds',
    help: 'FLAME model inference time',
    labelNames: ['expression_type'],
    buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1],
  }),

  avatar_expression_coefficients_updated_total: Counter({
    name: 'avatar_expression_coefficients_updated_total',
    help: 'Number of expression coefficient updates',
    labelNames: ['expression_type'],
  }),

  // Quality Metrics
  avatar_ssim_score: Gauge({
    name: 'avatar_ssim_score',
    help: 'Structural similarity index (0-1)',
    labelNames: ['avatar_id'],
  }),

  avatar_psnr_score: Gauge({
    name: 'avatar_psnr_score',
    help: 'Peak signal-to-noise ratio (dB)',
    labelNames: ['avatar_id'],
  }),
};

2.2 Rendering Performance Targets#

Metric Target Warning Critical
Frame Rate >= 30 FPS < 25 FPS < 15 FPS
Frame Render Time < 33ms > 40ms > 66ms
3DGS Render Latency < 20ms > 30ms > 50ms
FLAME Inference < 10ms > 20ms > 50ms
SSIM Score > 0.92 < 0.90 < 0.85

2.3 Dashboard Panel: Avatar Rendering#

json
{
  "title": "Avatar Rendering Performance",
  "type": "row",
  "panels": [
    {
      "title": "Frame Rate (FPS)",
      "type": "stat",
      "targets": [
        {
          "expr": "avg(avatar_frames_per_second)",
          "legendFormat": "Avg FPS"
        }
      ],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "red", "value": 0 },
          { "color": "yellow", "value": 25 },
          { "color": "green", "value": 30 }
        ]
      }
    },
    {
      "title": "Render Latency P95",
      "type": "stat",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(avatar_frame_render_duration_seconds_bucket[5m]))",
          "legendFormat": "P95"
        }
      ],
      "unit": "s"
    },
    {
      "title": "Frame Rate Over Time",
      "type": "timeseries",
      "targets": [
        {
          "expr": "avatar_frames_per_second",
          "legendFormat": "{{session_id}}"
        }
      ]
    },
    {
      "title": "Quality Score (SSIM)",
      "type": "gauge",
      "targets": [
        {
          "expr": "avg(avatar_ssim_score)",
          "legendFormat": "SSIM"
        }
      ],
      "min": 0,
      "max": 1
    }
  ]
}

3. Voice Pipeline Metrics#

3.1 Core Metrics#

typescript
// Voice Engine Metrics
const voiceMetrics = {
  // End-to-End Latency
  voice_e2e_latency_seconds: Histogram({
    name: 'voice_e2e_latency_seconds',
    help: 'End-to-end voice pipeline latency',
    labelNames: ['session_id'],
    buckets: [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 5],
  }),

  // Speech-to-Text
  voice_stt_latency_seconds: Histogram({
    name: 'voice_stt_latency_seconds',
    help: 'Speech-to-text processing latency',
    labelNames: ['provider', 'language'],
    buckets: [0.05, 0.1, 0.15, 0.2, 0.3, 0.5, 1],
  }),

  voice_stt_requests_total: Counter({
    name: 'voice_stt_requests_total',
    help: 'Total STT requests',
    labelNames: ['provider', 'status'],
  }),

  voice_stt_word_error_rate: Gauge({
    name: 'voice_stt_word_error_rate',
    help: 'Word error rate (0-1)',
    labelNames: ['provider'],
  }),

  // LLM Processing
  voice_llm_latency_seconds: Histogram({
    name: 'voice_llm_latency_seconds',
    help: 'LLM inference latency',
    labelNames: ['model', 'persona_id'],
    buckets: [0.1, 0.25, 0.5, 1, 2, 3, 5, 10],
  }),

  voice_llm_tokens_total: Counter({
    name: 'voice_llm_tokens_total',
    help: 'Total LLM tokens processed',
    labelNames: ['model', 'type'], // type: input, output
  }),

  // Text-to-Speech
  voice_tts_latency_seconds: Histogram({
    name: 'voice_tts_latency_seconds',
    help: 'Text-to-speech synthesis latency',
    labelNames: ['provider', 'voice_id'],
    buckets: [0.025, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5],
  }),

  voice_tts_requests_total: Counter({
    name: 'voice_tts_requests_total',
    help: 'Total TTS requests',
    labelNames: ['provider', 'status'],
  }),

  voice_tts_characters_total: Counter({
    name: 'voice_tts_characters_total',
    help: 'Total characters synthesized',
    labelNames: ['provider'],
  }),

  // Audio Quality
  voice_audio_quality_score: Gauge({
    name: 'voice_audio_quality_score',
    help: 'Audio quality score (0-1)',
    labelNames: ['session_id'],
  }),

  voice_lip_sync_accuracy: Gauge({
    name: 'voice_lip_sync_accuracy',
    help: 'Lip sync accuracy score (0-1)',
    labelNames: ['session_id'],
  }),
};

3.2 Voice Latency Targets#

Component Target Warning Critical
Speech-to-Text < 100ms > 150ms > 300ms
LLM Inference < 200ms > 500ms > 1000ms
Text-to-Speech < 100ms > 150ms > 300ms
End-to-End Total < 500ms > 750ms > 1500ms

3.3 Dashboard Panel: Voice Pipeline#

json
{
  "title": "Voice Pipeline Performance",
  "type": "row",
  "panels": [
    {
      "title": "E2E Latency P95",
      "type": "stat",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m]))",
          "legendFormat": "P95"
        }
      ],
      "unit": "s",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 0.5 },
          { "color": "red", "value": 1 }
        ]
      }
    },
    {
      "title": "Latency Breakdown",
      "type": "bargauge",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(voice_stt_latency_seconds_bucket[5m]))",
          "legendFormat": "STT"
        },
        {
          "expr": "histogram_quantile(0.95, rate(voice_llm_latency_seconds_bucket[5m]))",
          "legendFormat": "LLM"
        },
        {
          "expr": "histogram_quantile(0.95, rate(voice_tts_latency_seconds_bucket[5m]))",
          "legendFormat": "TTS"
        }
      ],
      "unit": "s"
    },
    {
      "title": "E2E Latency Distribution",
      "type": "heatmap",
      "targets": [
        {
          "expr": "rate(voice_e2e_latency_seconds_bucket[5m])",
          "format": "heatmap"
        }
      ]
    },
    {
      "title": "Audio Quality Score",
      "type": "gauge",
      "targets": [
        {
          "expr": "avg(voice_audio_quality_score)",
          "legendFormat": "Quality"
        }
      ],
      "min": 0,
      "max": 1
    }
  ]
}

4. Error Rate Metrics#

4.1 Core Metrics#

typescript
// Error Tracking Metrics
const errorMetrics = {
  // HTTP Errors
  http_requests_total: Counter({
    name: 'http_requests_total',
    help: 'Total HTTP requests',
    labelNames: ['service', 'method', 'path', 'status'],
  }),

  http_errors_total: Counter({
    name: 'http_errors_total',
    help: 'Total HTTP errors (4xx, 5xx)',
    labelNames: ['service', 'method', 'path', 'status', 'error_type'],
  }),

  // Service Errors
  service_errors_total: Counter({
    name: 'service_errors_total',
    help: 'Total service-level errors',
    labelNames: ['service', 'error_code', 'severity'],
  }),

  // Platform-Specific Errors
  avatar_errors_total: Counter({
    name: 'avatar_errors_total',
    help: 'Avatar rendering errors',
    labelNames: ['error_type', 'gpu_id'],
  }),

  voice_errors_total: Counter({
    name: 'voice_errors_total',
    help: 'Voice pipeline errors',
    labelNames: ['component', 'error_type', 'provider'],
  }),

  conferencing_errors_total: Counter({
    name: 'conferencing_errors_total',
    help: 'Video conferencing errors',
    labelNames: ['platform', 'error_type'],
  }),

  // Error Rate (computed)
  error_rate: Gauge({
    name: 'error_rate',
    help: 'Current error rate (0-1)',
    labelNames: ['service'],
  }),
};

4.2 Error Rate Targets#

Service Target Warning Critical
Avatar Engine < 0.1% > 0.5% > 1%
Voice Engine < 0.5% > 1% > 2%
Session Manager < 0.1% > 0.5% > 1%
Memory Service < 0.1% > 0.5% > 1%
Conferencing (Zoom/Teams) < 1% > 2% > 5%

4.3 Dashboard Panel: Error Rates#

json
{
  "title": "Error Rate Monitoring",
  "type": "row",
  "panels": [
    {
      "title": "Overall Error Rate",
      "type": "stat",
      "targets": [
        {
          "expr": "sum(rate(http_errors_total[5m])) / sum(rate(http_requests_total[5m])) * 100",
          "legendFormat": "Error %"
        }
      ],
      "unit": "percent",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 0.5 },
          { "color": "red", "value": 1 }
        ]
      }
    },
    {
      "title": "Error Rate by Service",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum by(service) (rate(http_errors_total[5m])) / sum by(service) (rate(http_requests_total[5m])) * 100",
          "legendFormat": "{{service}}"
        }
      ],
      "unit": "percent"
    },
    {
      "title": "Error Types Distribution",
      "type": "piechart",
      "targets": [
        {
          "expr": "sum by(error_type) (increase(service_errors_total[1h]))",
          "legendFormat": "{{error_type}}"
        }
      ]
    },
    {
      "title": "Recent Errors",
      "type": "table",
      "targets": [
        {
          "expr": "topk(10, sum by(service, error_code) (increase(service_errors_total[5m])))",
          "format": "table"
        }
      ]
    }
  ]
}

5. GPU Utilization Metrics#

5.1 Core Metrics#

typescript
// GPU Metrics (from @oshun/gpu-dispatcher)
const gpuMetrics = {
  // Utilization
  gpu_utilization_percent: Gauge({
    name: 'gpu_utilization_percent',
    help: 'GPU compute utilization percentage',
    labelNames: ['gpu_id', 'gpu_type'],
  }),

  gpu_memory_used_bytes: Gauge({
    name: 'gpu_memory_used_bytes',
    help: 'GPU memory used in bytes',
    labelNames: ['gpu_id', 'gpu_type'],
  }),

  gpu_memory_total_bytes: Gauge({
    name: 'gpu_memory_total_bytes',
    help: 'Total GPU memory in bytes',
    labelNames: ['gpu_id', 'gpu_type'],
  }),

  gpu_temperature_celsius: Gauge({
    name: 'gpu_temperature_celsius',
    help: 'GPU temperature in Celsius',
    labelNames: ['gpu_id'],
  }),

  gpu_power_watts: Gauge({
    name: 'gpu_power_watts',
    help: 'GPU power consumption in watts',
    labelNames: ['gpu_id'],
  }),

  // Job Processing
  gpu_jobs_total: Counter({
    name: 'gpu_jobs_total',
    help: 'Total GPU jobs submitted',
    labelNames: ['job_type', 'priority'],
  }),

  gpu_jobs_completed_total: Counter({
    name: 'gpu_jobs_completed_total',
    help: 'Completed GPU jobs',
    labelNames: ['job_type', 'status'],
  }),

  gpu_jobs_pending: Gauge({
    name: 'gpu_jobs_pending',
    help: 'Jobs waiting in queue',
    labelNames: ['job_type'],
  }),

  gpu_jobs_running: Gauge({
    name: 'gpu_jobs_running',
    help: 'Currently running jobs',
    labelNames: ['gpu_id'],
  }),

  gpu_job_duration_seconds: Histogram({
    name: 'gpu_job_duration_seconds',
    help: 'GPU job execution duration',
    labelNames: ['job_type'],
    buckets: [0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600],
  }),

  gpu_queue_wait_seconds: Histogram({
    name: 'gpu_queue_wait_seconds',
    help: 'Time spent waiting in queue',
    labelNames: ['job_type', 'priority'],
    buckets: [0.1, 0.5, 1, 2, 5, 10, 30, 60, 120],
  }),

  // Endpoint Health
  gpu_endpoint_health: Gauge({
    name: 'gpu_endpoint_health',
    help: 'Endpoint health (0=unhealthy, 0.5=degraded, 1=healthy)',
    labelNames: ['endpoint_id', 'gpu_type'],
  }),

  gpu_circuit_breaker_state: Gauge({
    name: 'gpu_circuit_breaker_state',
    help: 'Circuit breaker state (0=closed, 0.5=half-open, 1=open)',
    labelNames: ['endpoint_id'],
  }),

  // Cost Tracking
  gpu_cost_cents_total: Counter({
    name: 'gpu_cost_cents_total',
    help: 'Total GPU cost in cents',
    labelNames: ['gpu_type', 'job_type'],
  }),

  gpu_seconds_total: Counter({
    name: 'gpu_seconds_total',
    help: 'Total GPU seconds consumed',
    labelNames: ['gpu_type'],
  }),
};

5.2 GPU Utilization Targets#

Metric Target Warning Critical
GPU Utilization 60-80% > 90% > 95%
Memory Utilization 60-80% > 85% > 95%
Temperature < 75°C > 80°C > 85°C
Queue Depth < 10 > 50 > 100
Queue Wait Time (P95) < 5s > 30s > 60s

5.3 Dashboard Panel: GPU Utilization#

json
{
  "title": "GPU Utilization",
  "type": "row",
  "panels": [
    {
      "title": "GPU Pool Status",
      "type": "stat",
      "targets": [
        {
          "expr": "count(gpu_endpoint_health == 1)",
          "legendFormat": "Healthy"
        },
        {
          "expr": "count(gpu_endpoint_health == 0.5)",
          "legendFormat": "Degraded"
        },
        {
          "expr": "count(gpu_endpoint_health == 0)",
          "legendFormat": "Unhealthy"
        }
      ]
    },
    {
      "title": "GPU Utilization by Node",
      "type": "timeseries",
      "targets": [
        {
          "expr": "gpu_utilization_percent",
          "legendFormat": "{{gpu_id}}"
        }
      ],
      "unit": "percent"
    },
    {
      "title": "GPU Memory Usage",
      "type": "gauge",
      "targets": [
        {
          "expr": "sum(gpu_memory_used_bytes) / sum(gpu_memory_total_bytes) * 100",
          "legendFormat": "Memory %"
        }
      ],
      "unit": "percent",
      "max": 100
    },
    {
      "title": "Job Queue Depth",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum(gpu_jobs_pending)",
          "legendFormat": "Pending"
        },
        {
          "expr": "sum(gpu_jobs_running)",
          "legendFormat": "Running"
        }
      ]
    },
    {
      "title": "GPU Temperature",
      "type": "timeseries",
      "targets": [
        {
          "expr": "gpu_temperature_celsius",
          "legendFormat": "{{gpu_id}}"
        }
      ],
      "unit": "celsius",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 75 },
          { "color": "red", "value": 85 }
        ]
      }
    },
    {
      "title": "Circuit Breaker Status",
      "type": "table",
      "targets": [
        {
          "expr": "gpu_circuit_breaker_state",
          "format": "table"
        }
      ]
    }
  ]
}

6. Alert Rules Configuration#

6.1 Avatar Rendering Alerts#

yaml
groups:
  - name: psyche-avatar-alerts
    rules:
      - alert: AvatarLowFPS
        expr: avatar_frames_per_second < 25
        for: 2m
        labels:
          severity: warning
          service: avatar-engine
        annotations:
          summary: 'Avatar rendering FPS below threshold'
          description:
            'FPS is {{ $value }} for session {{ $labels.session_id }}'

      - alert: AvatarCriticalFPS
        expr: avatar_frames_per_second < 15
        for: 1m
        labels:
          severity: critical
          service: avatar-engine
        annotations:
          summary: 'Avatar rendering FPS critically low'
          description:
            'FPS is {{ $value }} for session {{ $labels.session_id }}'

      - alert: AvatarRenderLatencyHigh
        expr:
          histogram_quantile(0.95,
          rate(avatar_frame_render_duration_seconds_bucket[5m])) > 0.04
        for: 3m
        labels:
          severity: warning
          service: avatar-engine
        annotations:
          summary: 'Avatar render latency P95 above 40ms'
          description: 'P95 latency is {{ $value }}s'

      - alert: AvatarQualityDegraded
        expr: avg(avatar_ssim_score) < 0.90
        for: 5m
        labels:
          severity: warning
          service: avatar-engine
        annotations:
          summary: 'Avatar rendering quality degraded'
          description: 'SSIM score is {{ $value }}'

6.2 Voice Pipeline Alerts#

yaml
groups:
  - name: psyche-voice-alerts
    rules:
      - alert: VoiceE2ELatencyHigh
        expr:
          histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m])) >
          0.75
        for: 3m
        labels:
          severity: warning
          service: voice-engine
        annotations:
          summary: 'Voice E2E latency P95 above 750ms'
          description: 'P95 latency is {{ $value }}s'

      - alert: VoiceE2ELatencyCritical
        expr:
          histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m])) >
          1.5
        for: 1m
        labels:
          severity: critical
          service: voice-engine
        annotations:
          summary: 'Voice E2E latency P95 above 1.5s'
          description: 'P95 latency is {{ $value }}s'

      - alert: STTLatencyHigh
        expr:
          histogram_quantile(0.95, rate(voice_stt_latency_seconds_bucket[5m])) >
          0.15
        for: 3m
        labels:
          severity: warning
          service: voice-engine
          component: stt
        annotations:
          summary: 'STT latency P95 above 150ms'
          description: 'P95 latency is {{ $value }}s'

      - alert: TTSLatencyHigh
        expr:
          histogram_quantile(0.95, rate(voice_tts_latency_seconds_bucket[5m])) >
          0.15
        for: 3m
        labels:
          severity: warning
          service: voice-engine
          component: tts
        annotations:
          summary: 'TTS latency P95 above 150ms'
          description: 'P95 latency is {{ $value }}s'

6.3 Error Rate Alerts#

yaml
groups:
  - name: psyche-error-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_errors_total[5m]))
          / sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Error rate above 1%'
          description: 'Error rate is {{ $value | humanizePercentage }}'

      - alert: CriticalErrorRate
        expr: |
          sum(rate(http_errors_total[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'Error rate above 5%'
          description: 'Error rate is {{ $value | humanizePercentage }}'

      - alert: AvatarErrorsIncreasing
        expr: rate(avatar_errors_total[5m]) > 1
        for: 3m
        labels:
          severity: warning
          service: avatar-engine
        annotations:
          summary: 'Avatar errors increasing'
          description: 'Error rate is {{ $value }}/s'

      - alert: VoiceErrorsIncreasing
        expr: rate(voice_errors_total[5m]) > 1
        for: 3m
        labels:
          severity: warning
          service: voice-engine
        annotations:
          summary: 'Voice pipeline errors increasing'
          description: 'Error rate is {{ $value }}/s'

6.4 GPU Alerts#

yaml
groups:
  - name: psyche-gpu-alerts
    rules:
      - alert: GPUHighUtilization
        expr: gpu_utilization_percent > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'GPU utilization above 90%'
          description: 'GPU {{ $labels.gpu_id }} at {{ $value }}%'

      - alert: GPUCriticalUtilization
        expr: gpu_utilization_percent > 95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'GPU utilization above 95%'
          description: 'GPU {{ $labels.gpu_id }} at {{ $value }}%'

      - alert: GPUMemoryExhaustion
        expr: gpu_memory_used_bytes / gpu_memory_total_bytes > 0.95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'GPU memory nearly exhausted'
          description:
            'GPU {{ $labels.gpu_id }} memory at {{ $value | humanizePercentage
            }}'

      - alert: GPUOverheating
        expr: gpu_temperature_celsius > 85
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'GPU overheating'
          description: 'GPU {{ $labels.gpu_id }} at {{ $value }}°C'

      - alert: GPUQueueBacklog
        expr: sum(gpu_jobs_pending) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'GPU job queue backlog'
          description: '{{ $value }} jobs pending'

      - alert: GPUCircuitBreakerOpen
        expr: gpu_circuit_breaker_state == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'GPU circuit breaker open'
          description: 'Circuit breaker open for {{ $labels.endpoint_id }}'

      - alert: GPUEndpointUnhealthy
        expr: gpu_endpoint_health == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'GPU endpoint unhealthy'
          description: 'Endpoint {{ $labels.endpoint_id }} is unhealthy'

7. Grafana Dashboard Configuration#

7.1 Dashboard Manifest#

json
{
  "dashboard": {
    "title": "Psyche Platform - Launch Monitoring",
    "uid": "psyche-launch-monitoring",
    "tags": ["psyche", "launch", "production"],
    "timezone": "browser",
    "refresh": "10s",
    "rows": [
      {
        "title": "Overview",
        "panels": [
          "service_health",
          "active_sessions",
          "error_rate",
          "p95_latency"
        ]
      },
      {
        "title": "Avatar Rendering",
        "panels": ["fps", "render_latency", "quality_score", "gpu_memory"]
      },
      {
        "title": "Voice Pipeline",
        "panels": [
          "e2e_latency",
          "latency_breakdown",
          "stt_metrics",
          "tts_metrics"
        ]
      },
      {
        "title": "Error Tracking",
        "panels": [
          "error_rate_by_service",
          "error_distribution",
          "recent_errors"
        ]
      },
      {
        "title": "GPU Utilization",
        "panels": [
          "gpu_pool_status",
          "gpu_utilization",
          "job_queue",
          "circuit_breakers"
        ]
      },
      {
        "title": "Infrastructure",
        "panels": ["database_pool", "cache_hit_rate", "rate_limiter_status"]
      }
    ]
  }
}

7.2 Key Dashboard Variables#

json
{
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "query",
        "query": "label_values(up, service)",
        "multi": true
      },
      {
        "name": "gpu_id",
        "type": "query",
        "query": "label_values(gpu_utilization_percent, gpu_id)",
        "multi": true
      },
      {
        "name": "session_id",
        "type": "query",
        "query": "label_values(avatar_frames_per_second, session_id)",
        "multi": false
      }
    ]
  }
}

8. AlertManager Configuration#

8.1 Routing Configuration#

yaml
# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: '${SLACK_WEBHOOK_URL}'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

route:
  group_by: ['alertname', 'service', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'psyche-default'
  routes:
    # Critical alerts -> PagerDuty + Slack
    - match:
        severity: critical
      receiver: 'psyche-critical'
      continue: true

    # Warning alerts -> Slack only
    - match:
        severity: warning
      receiver: 'psyche-warnings'

    # GPU-specific alerts
    - match:
        service: gpu
      receiver: 'psyche-gpu-team'

receivers:
  - name: 'psyche-default'
    slack_configs:
      - channel: '#psyche-alerts'
        send_resolved: true

  - name: 'psyche-critical'
    pagerduty_configs:
      - service_key: '${PAGERDUTY_SERVICE_KEY}'
        severity: critical
    slack_configs:
      - channel: '#psyche-critical'
        send_resolved: true

  - name: 'psyche-warnings'
    slack_configs:
      - channel: '#psyche-warnings'
        send_resolved: true

  - name: 'psyche-gpu-team'
    slack_configs:
      - channel: '#psyche-gpu-ops'
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'service']

9. Launch Readiness Checklist#

9.1 Monitoring Components#

Component Status Notes
Prometheus Deployed READY Scraping all services
Grafana Dashboards READY 6 dashboard panels configured
Alert Rules READY 15+ alert rules configured
AlertManager READY PagerDuty + Slack integration
GPU Metrics Collector READY All GPU nodes reporting
Voice Pipeline Metrics READY E2E latency tracking active
Avatar Rendering Metrics READY FPS and quality tracking active
Error Rate Tracking READY Per-service error rates

9.2 SLO Targets#

SLO Target Measurement
Availability 99.9% Uptime over 30 days
Avatar FPS >= 30 FPS P95 over 1 hour
Voice E2E Latency < 500ms P95 over 1 hour
Error Rate < 0.1% Rolling 1 hour average
GPU Job Queue Time < 5s P95 over 1 hour

10. Sign-off#

Role Name Date Signature
SRE Lead _ ____ __
Platform Lead _ ____ __
Security Lead _ ____ __
VP Engineering _ ____ __

Configuration generated: 2026-02-01 Platform: Psyche AI Virtual Assistant Document classification: Internal - Operations