# Psyche Platform - Launch Monitoring Configuration

**Document Version:** 1.0 **Configuration Date:** 2026-02-01 **Platform:**
Psyche AI Virtual Assistant **Status:** OPERATIONAL - Monitoring dashboard ready

---

## Executive Summary

This document provides the comprehensive monitoring configuration for the Psyche
AI Virtual Assistant Platform launch. The monitoring stack leverages the
existing `@oshun/metrics`, `@oshun/tracing`, and domain-specific metrics
collectors to provide real-time visibility into avatar rendering, voice
pipeline, error rates, and GPU utilization.

### Monitoring Stack Overview

| Component           | Technology    | Purpose                       |
| ------------------- | ------------- | ----------------------------- |
| Metrics Collection  | prom-client   | Prometheus-compatible metrics |
| Distributed Tracing | OpenTelemetry | Request tracing               |
| Metrics Storage     | Prometheus    | Time-series database          |
| Visualization       | Grafana       | Dashboards and alerts         |
| Alerting            | AlertManager  | Alert routing                 |

---

## 1. Metrics Architecture

### 1.1 Metrics Endpoint Configuration

```yaml
# Service Metrics Exposure
services:
  avatar-engine:
    metrics_port: 9090
    metrics_path: /metrics
    health_path: /health
    ready_path: /ready

  voice-engine:
    metrics_port: 9091
    metrics_path: /metrics
    health_path: /health
    ready_path: /ready

  session-manager:
    metrics_port: 9092
    metrics_path: /metrics
    health_path: /health
    ready_path: /ready

  memory-service:
    metrics_port: 9093
    metrics_path: /metrics
    health_path: /health
    ready_path: /ready
```

### 1.2 Prometheus Scrape Configuration

```yaml
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Psyche Avatar Engine
  - job_name: 'psyche-avatar-engine'
    static_configs:
      - targets: ['avatar-engine:9090']
    relabel_configs:
      - source_labels: [__address__]
        target_label: service
        replacement: 'avatar-engine'

  # Psyche Voice Engine
  - job_name: 'psyche-voice-engine'
    static_configs:
      - targets: ['voice-engine:9091']
    relabel_configs:
      - source_labels: [__address__]
        target_label: service
        replacement: 'voice-engine'

  # Psyche Session Manager
  - job_name: 'psyche-session-manager'
    static_configs:
      - targets: ['session-manager:9092']
    relabel_configs:
      - source_labels: [__address__]
        target_label: service
        replacement: 'session-manager'

  # Psyche Memory Service
  - job_name: 'psyche-memory-service'
    static_configs:
      - targets: ['memory-service:9093']
    relabel_configs:
      - source_labels: [__address__]
        target_label: service
        replacement: 'memory-service'

  # GPU Endpoints (RunPod/Cloud)
  - job_name: 'psyche-gpu-endpoints'
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names: ['psyche-gpu']
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_app]
        action: keep
        regex: psyche-gpu-.*
```

---

## 2. Avatar Rendering Metrics

### 2.1 Core Metrics

```typescript
// Avatar Engine Metrics
const avatarMetrics = {
  // Frame Performance
  avatar_frame_render_duration_seconds: Histogram({
    name: 'avatar_frame_render_duration_seconds',
    help: 'Time to render a single avatar frame',
    labelNames: ['resolution', 'quality', 'gpu_type'],
    buckets: [0.005, 0.01, 0.016, 0.025, 0.033, 0.05, 0.1, 0.25],
  }),

  avatar_frames_per_second: Gauge({
    name: 'avatar_frames_per_second',
    help: 'Current avatar rendering FPS',
    labelNames: ['session_id', 'quality'],
  }),

  // 3D Gaussian Splatting
  avatar_3dgs_points_total: Gauge({
    name: 'avatar_3dgs_points_total',
    help: 'Number of 3DGS points in current model',
    labelNames: ['avatar_id'],
  }),

  avatar_3dgs_render_latency_seconds: Histogram({
    name: 'avatar_3dgs_render_latency_seconds',
    help: '3DGS rendering latency',
    labelNames: ['resolution'],
    buckets: [0.005, 0.01, 0.02, 0.033, 0.05, 0.1],
  }),

  // FLAME Expression
  avatar_flame_inference_duration_seconds: Histogram({
    name: 'avatar_flame_inference_duration_seconds',
    help: 'FLAME model inference time',
    labelNames: ['expression_type'],
    buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1],
  }),

  avatar_expression_coefficients_updated_total: Counter({
    name: 'avatar_expression_coefficients_updated_total',
    help: 'Number of expression coefficient updates',
    labelNames: ['expression_type'],
  }),

  // Quality Metrics
  avatar_ssim_score: Gauge({
    name: 'avatar_ssim_score',
    help: 'Structural similarity index (0-1)',
    labelNames: ['avatar_id'],
  }),

  avatar_psnr_score: Gauge({
    name: 'avatar_psnr_score',
    help: 'Peak signal-to-noise ratio (dB)',
    labelNames: ['avatar_id'],
  }),
};
```

### 2.2 Rendering Performance Targets

| Metric              | Target    | Warning  | Critical |
| ------------------- | --------- | -------- | -------- |
| Frame Rate          | >= 30 FPS | < 25 FPS | < 15 FPS |
| Frame Render Time   | < 33ms    | > 40ms   | > 66ms   |
| 3DGS Render Latency | < 20ms    | > 30ms   | > 50ms   |
| FLAME Inference     | < 10ms    | > 20ms   | > 50ms   |
| SSIM Score          | > 0.92    | < 0.90   | < 0.85   |

### 2.3 Dashboard Panel: Avatar Rendering

```json
{
  "title": "Avatar Rendering Performance",
  "type": "row",
  "panels": [
    {
      "title": "Frame Rate (FPS)",
      "type": "stat",
      "targets": [
        {
          "expr": "avg(avatar_frames_per_second)",
          "legendFormat": "Avg FPS"
        }
      ],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "red", "value": 0 },
          { "color": "yellow", "value": 25 },
          { "color": "green", "value": 30 }
        ]
      }
    },
    {
      "title": "Render Latency P95",
      "type": "stat",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(avatar_frame_render_duration_seconds_bucket[5m]))",
          "legendFormat": "P95"
        }
      ],
      "unit": "s"
    },
    {
      "title": "Frame Rate Over Time",
      "type": "timeseries",
      "targets": [
        {
          "expr": "avatar_frames_per_second",
          "legendFormat": "{{session_id}}"
        }
      ]
    },
    {
      "title": "Quality Score (SSIM)",
      "type": "gauge",
      "targets": [
        {
          "expr": "avg(avatar_ssim_score)",
          "legendFormat": "SSIM"
        }
      ],
      "min": 0,
      "max": 1
    }
  ]
}
```

---

## 3. Voice Pipeline Metrics

### 3.1 Core Metrics

```typescript
// Voice Engine Metrics
const voiceMetrics = {
  // End-to-End Latency
  voice_e2e_latency_seconds: Histogram({
    name: 'voice_e2e_latency_seconds',
    help: 'End-to-end voice pipeline latency',
    labelNames: ['session_id'],
    buckets: [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 5],
  }),

  // Speech-to-Text
  voice_stt_latency_seconds: Histogram({
    name: 'voice_stt_latency_seconds',
    help: 'Speech-to-text processing latency',
    labelNames: ['provider', 'language'],
    buckets: [0.05, 0.1, 0.15, 0.2, 0.3, 0.5, 1],
  }),

  voice_stt_requests_total: Counter({
    name: 'voice_stt_requests_total',
    help: 'Total STT requests',
    labelNames: ['provider', 'status'],
  }),

  voice_stt_word_error_rate: Gauge({
    name: 'voice_stt_word_error_rate',
    help: 'Word error rate (0-1)',
    labelNames: ['provider'],
  }),

  // LLM Processing
  voice_llm_latency_seconds: Histogram({
    name: 'voice_llm_latency_seconds',
    help: 'LLM inference latency',
    labelNames: ['model', 'persona_id'],
    buckets: [0.1, 0.25, 0.5, 1, 2, 3, 5, 10],
  }),

  voice_llm_tokens_total: Counter({
    name: 'voice_llm_tokens_total',
    help: 'Total LLM tokens processed',
    labelNames: ['model', 'type'], // type: input, output
  }),

  // Text-to-Speech
  voice_tts_latency_seconds: Histogram({
    name: 'voice_tts_latency_seconds',
    help: 'Text-to-speech synthesis latency',
    labelNames: ['provider', 'voice_id'],
    buckets: [0.025, 0.05, 0.1, 0.15, 0.2, 0.3, 0.5],
  }),

  voice_tts_requests_total: Counter({
    name: 'voice_tts_requests_total',
    help: 'Total TTS requests',
    labelNames: ['provider', 'status'],
  }),

  voice_tts_characters_total: Counter({
    name: 'voice_tts_characters_total',
    help: 'Total characters synthesized',
    labelNames: ['provider'],
  }),

  // Audio Quality
  voice_audio_quality_score: Gauge({
    name: 'voice_audio_quality_score',
    help: 'Audio quality score (0-1)',
    labelNames: ['session_id'],
  }),

  voice_lip_sync_accuracy: Gauge({
    name: 'voice_lip_sync_accuracy',
    help: 'Lip sync accuracy score (0-1)',
    labelNames: ['session_id'],
  }),
};
```

### 3.2 Voice Latency Targets

| Component            | Target      | Warning     | Critical     |
| -------------------- | ----------- | ----------- | ------------ |
| Speech-to-Text       | < 100ms     | > 150ms     | > 300ms      |
| LLM Inference        | < 200ms     | > 500ms     | > 1000ms     |
| Text-to-Speech       | < 100ms     | > 150ms     | > 300ms      |
| **End-to-End Total** | **< 500ms** | **> 750ms** | **> 1500ms** |

### 3.3 Dashboard Panel: Voice Pipeline

```json
{
  "title": "Voice Pipeline Performance",
  "type": "row",
  "panels": [
    {
      "title": "E2E Latency P95",
      "type": "stat",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m]))",
          "legendFormat": "P95"
        }
      ],
      "unit": "s",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 0.5 },
          { "color": "red", "value": 1 }
        ]
      }
    },
    {
      "title": "Latency Breakdown",
      "type": "bargauge",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(voice_stt_latency_seconds_bucket[5m]))",
          "legendFormat": "STT"
        },
        {
          "expr": "histogram_quantile(0.95, rate(voice_llm_latency_seconds_bucket[5m]))",
          "legendFormat": "LLM"
        },
        {
          "expr": "histogram_quantile(0.95, rate(voice_tts_latency_seconds_bucket[5m]))",
          "legendFormat": "TTS"
        }
      ],
      "unit": "s"
    },
    {
      "title": "E2E Latency Distribution",
      "type": "heatmap",
      "targets": [
        {
          "expr": "rate(voice_e2e_latency_seconds_bucket[5m])",
          "format": "heatmap"
        }
      ]
    },
    {
      "title": "Audio Quality Score",
      "type": "gauge",
      "targets": [
        {
          "expr": "avg(voice_audio_quality_score)",
          "legendFormat": "Quality"
        }
      ],
      "min": 0,
      "max": 1
    }
  ]
}
```

---

## 4. Error Rate Metrics

### 4.1 Core Metrics

```typescript
// Error Tracking Metrics
const errorMetrics = {
  // HTTP Errors
  http_requests_total: Counter({
    name: 'http_requests_total',
    help: 'Total HTTP requests',
    labelNames: ['service', 'method', 'path', 'status'],
  }),

  http_errors_total: Counter({
    name: 'http_errors_total',
    help: 'Total HTTP errors (4xx, 5xx)',
    labelNames: ['service', 'method', 'path', 'status', 'error_type'],
  }),

  // Service Errors
  service_errors_total: Counter({
    name: 'service_errors_total',
    help: 'Total service-level errors',
    labelNames: ['service', 'error_code', 'severity'],
  }),

  // Platform-Specific Errors
  avatar_errors_total: Counter({
    name: 'avatar_errors_total',
    help: 'Avatar rendering errors',
    labelNames: ['error_type', 'gpu_id'],
  }),

  voice_errors_total: Counter({
    name: 'voice_errors_total',
    help: 'Voice pipeline errors',
    labelNames: ['component', 'error_type', 'provider'],
  }),

  conferencing_errors_total: Counter({
    name: 'conferencing_errors_total',
    help: 'Video conferencing errors',
    labelNames: ['platform', 'error_type'],
  }),

  // Error Rate (computed)
  error_rate: Gauge({
    name: 'error_rate',
    help: 'Current error rate (0-1)',
    labelNames: ['service'],
  }),
};
```

### 4.2 Error Rate Targets

| Service                   | Target | Warning | Critical |
| ------------------------- | ------ | ------- | -------- |
| Avatar Engine             | < 0.1% | > 0.5%  | > 1%     |
| Voice Engine              | < 0.5% | > 1%    | > 2%     |
| Session Manager           | < 0.1% | > 0.5%  | > 1%     |
| Memory Service            | < 0.1% | > 0.5%  | > 1%     |
| Conferencing (Zoom/Teams) | < 1%   | > 2%    | > 5%     |

### 4.3 Dashboard Panel: Error Rates

```json
{
  "title": "Error Rate Monitoring",
  "type": "row",
  "panels": [
    {
      "title": "Overall Error Rate",
      "type": "stat",
      "targets": [
        {
          "expr": "sum(rate(http_errors_total[5m])) / sum(rate(http_requests_total[5m])) * 100",
          "legendFormat": "Error %"
        }
      ],
      "unit": "percent",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 0.5 },
          { "color": "red", "value": 1 }
        ]
      }
    },
    {
      "title": "Error Rate by Service",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum by(service) (rate(http_errors_total[5m])) / sum by(service) (rate(http_requests_total[5m])) * 100",
          "legendFormat": "{{service}}"
        }
      ],
      "unit": "percent"
    },
    {
      "title": "Error Types Distribution",
      "type": "piechart",
      "targets": [
        {
          "expr": "sum by(error_type) (increase(service_errors_total[1h]))",
          "legendFormat": "{{error_type}}"
        }
      ]
    },
    {
      "title": "Recent Errors",
      "type": "table",
      "targets": [
        {
          "expr": "topk(10, sum by(service, error_code) (increase(service_errors_total[5m])))",
          "format": "table"
        }
      ]
    }
  ]
}
```

---

## 5. GPU Utilization Metrics

### 5.1 Core Metrics

```typescript
// GPU Metrics (from @oshun/gpu-dispatcher)
const gpuMetrics = {
  // Utilization
  gpu_utilization_percent: Gauge({
    name: 'gpu_utilization_percent',
    help: 'GPU compute utilization percentage',
    labelNames: ['gpu_id', 'gpu_type'],
  }),

  gpu_memory_used_bytes: Gauge({
    name: 'gpu_memory_used_bytes',
    help: 'GPU memory used in bytes',
    labelNames: ['gpu_id', 'gpu_type'],
  }),

  gpu_memory_total_bytes: Gauge({
    name: 'gpu_memory_total_bytes',
    help: 'Total GPU memory in bytes',
    labelNames: ['gpu_id', 'gpu_type'],
  }),

  gpu_temperature_celsius: Gauge({
    name: 'gpu_temperature_celsius',
    help: 'GPU temperature in Celsius',
    labelNames: ['gpu_id'],
  }),

  gpu_power_watts: Gauge({
    name: 'gpu_power_watts',
    help: 'GPU power consumption in watts',
    labelNames: ['gpu_id'],
  }),

  // Job Processing
  gpu_jobs_total: Counter({
    name: 'gpu_jobs_total',
    help: 'Total GPU jobs submitted',
    labelNames: ['job_type', 'priority'],
  }),

  gpu_jobs_completed_total: Counter({
    name: 'gpu_jobs_completed_total',
    help: 'Completed GPU jobs',
    labelNames: ['job_type', 'status'],
  }),

  gpu_jobs_pending: Gauge({
    name: 'gpu_jobs_pending',
    help: 'Jobs waiting in queue',
    labelNames: ['job_type'],
  }),

  gpu_jobs_running: Gauge({
    name: 'gpu_jobs_running',
    help: 'Currently running jobs',
    labelNames: ['gpu_id'],
  }),

  gpu_job_duration_seconds: Histogram({
    name: 'gpu_job_duration_seconds',
    help: 'GPU job execution duration',
    labelNames: ['job_type'],
    buckets: [0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600],
  }),

  gpu_queue_wait_seconds: Histogram({
    name: 'gpu_queue_wait_seconds',
    help: 'Time spent waiting in queue',
    labelNames: ['job_type', 'priority'],
    buckets: [0.1, 0.5, 1, 2, 5, 10, 30, 60, 120],
  }),

  // Endpoint Health
  gpu_endpoint_health: Gauge({
    name: 'gpu_endpoint_health',
    help: 'Endpoint health (0=unhealthy, 0.5=degraded, 1=healthy)',
    labelNames: ['endpoint_id', 'gpu_type'],
  }),

  gpu_circuit_breaker_state: Gauge({
    name: 'gpu_circuit_breaker_state',
    help: 'Circuit breaker state (0=closed, 0.5=half-open, 1=open)',
    labelNames: ['endpoint_id'],
  }),

  // Cost Tracking
  gpu_cost_cents_total: Counter({
    name: 'gpu_cost_cents_total',
    help: 'Total GPU cost in cents',
    labelNames: ['gpu_type', 'job_type'],
  }),

  gpu_seconds_total: Counter({
    name: 'gpu_seconds_total',
    help: 'Total GPU seconds consumed',
    labelNames: ['gpu_type'],
  }),
};
```

### 5.2 GPU Utilization Targets

| Metric                | Target | Warning | Critical |
| --------------------- | ------ | ------- | -------- |
| GPU Utilization       | 60-80% | > 90%   | > 95%    |
| Memory Utilization    | 60-80% | > 85%   | > 95%    |
| Temperature           | < 75°C | > 80°C  | > 85°C   |
| Queue Depth           | < 10   | > 50    | > 100    |
| Queue Wait Time (P95) | < 5s   | > 30s   | > 60s    |

### 5.3 Dashboard Panel: GPU Utilization

```json
{
  "title": "GPU Utilization",
  "type": "row",
  "panels": [
    {
      "title": "GPU Pool Status",
      "type": "stat",
      "targets": [
        {
          "expr": "count(gpu_endpoint_health == 1)",
          "legendFormat": "Healthy"
        },
        {
          "expr": "count(gpu_endpoint_health == 0.5)",
          "legendFormat": "Degraded"
        },
        {
          "expr": "count(gpu_endpoint_health == 0)",
          "legendFormat": "Unhealthy"
        }
      ]
    },
    {
      "title": "GPU Utilization by Node",
      "type": "timeseries",
      "targets": [
        {
          "expr": "gpu_utilization_percent",
          "legendFormat": "{{gpu_id}}"
        }
      ],
      "unit": "percent"
    },
    {
      "title": "GPU Memory Usage",
      "type": "gauge",
      "targets": [
        {
          "expr": "sum(gpu_memory_used_bytes) / sum(gpu_memory_total_bytes) * 100",
          "legendFormat": "Memory %"
        }
      ],
      "unit": "percent",
      "max": 100
    },
    {
      "title": "Job Queue Depth",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum(gpu_jobs_pending)",
          "legendFormat": "Pending"
        },
        {
          "expr": "sum(gpu_jobs_running)",
          "legendFormat": "Running"
        }
      ]
    },
    {
      "title": "GPU Temperature",
      "type": "timeseries",
      "targets": [
        {
          "expr": "gpu_temperature_celsius",
          "legendFormat": "{{gpu_id}}"
        }
      ],
      "unit": "celsius",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": 0 },
          { "color": "yellow", "value": 75 },
          { "color": "red", "value": 85 }
        ]
      }
    },
    {
      "title": "Circuit Breaker Status",
      "type": "table",
      "targets": [
        {
          "expr": "gpu_circuit_breaker_state",
          "format": "table"
        }
      ]
    }
  ]
}
```

---

## 6. Alert Rules Configuration

### 6.1 Avatar Rendering Alerts

```yaml
groups:
  - name: psyche-avatar-alerts
    rules:
      - alert: AvatarLowFPS
        expr: avatar_frames_per_second < 25
        for: 2m
        labels:
          severity: warning
          service: avatar-engine
        annotations:
          summary: 'Avatar rendering FPS below threshold'
          description:
            'FPS is {{ $value }} for session {{ $labels.session_id }}'

      - alert: AvatarCriticalFPS
        expr: avatar_frames_per_second < 15
        for: 1m
        labels:
          severity: critical
          service: avatar-engine
        annotations:
          summary: 'Avatar rendering FPS critically low'
          description:
            'FPS is {{ $value }} for session {{ $labels.session_id }}'

      - alert: AvatarRenderLatencyHigh
        expr:
          histogram_quantile(0.95,
          rate(avatar_frame_render_duration_seconds_bucket[5m])) > 0.04
        for: 3m
        labels:
          severity: warning
          service: avatar-engine
        annotations:
          summary: 'Avatar render latency P95 above 40ms'
          description: 'P95 latency is {{ $value }}s'

      - alert: AvatarQualityDegraded
        expr: avg(avatar_ssim_score) < 0.90
        for: 5m
        labels:
          severity: warning
          service: avatar-engine
        annotations:
          summary: 'Avatar rendering quality degraded'
          description: 'SSIM score is {{ $value }}'
```

### 6.2 Voice Pipeline Alerts

```yaml
groups:
  - name: psyche-voice-alerts
    rules:
      - alert: VoiceE2ELatencyHigh
        expr:
          histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m])) >
          0.75
        for: 3m
        labels:
          severity: warning
          service: voice-engine
        annotations:
          summary: 'Voice E2E latency P95 above 750ms'
          description: 'P95 latency is {{ $value }}s'

      - alert: VoiceE2ELatencyCritical
        expr:
          histogram_quantile(0.95, rate(voice_e2e_latency_seconds_bucket[5m])) >
          1.5
        for: 1m
        labels:
          severity: critical
          service: voice-engine
        annotations:
          summary: 'Voice E2E latency P95 above 1.5s'
          description: 'P95 latency is {{ $value }}s'

      - alert: STTLatencyHigh
        expr:
          histogram_quantile(0.95, rate(voice_stt_latency_seconds_bucket[5m])) >
          0.15
        for: 3m
        labels:
          severity: warning
          service: voice-engine
          component: stt
        annotations:
          summary: 'STT latency P95 above 150ms'
          description: 'P95 latency is {{ $value }}s'

      - alert: TTSLatencyHigh
        expr:
          histogram_quantile(0.95, rate(voice_tts_latency_seconds_bucket[5m])) >
          0.15
        for: 3m
        labels:
          severity: warning
          service: voice-engine
          component: tts
        annotations:
          summary: 'TTS latency P95 above 150ms'
          description: 'P95 latency is {{ $value }}s'
```

### 6.3 Error Rate Alerts

```yaml
groups:
  - name: psyche-error-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_errors_total[5m]))
          / sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Error rate above 1%'
          description: 'Error rate is {{ $value | humanizePercentage }}'

      - alert: CriticalErrorRate
        expr: |
          sum(rate(http_errors_total[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'Error rate above 5%'
          description: 'Error rate is {{ $value | humanizePercentage }}'

      - alert: AvatarErrorsIncreasing
        expr: rate(avatar_errors_total[5m]) > 1
        for: 3m
        labels:
          severity: warning
          service: avatar-engine
        annotations:
          summary: 'Avatar errors increasing'
          description: 'Error rate is {{ $value }}/s'

      - alert: VoiceErrorsIncreasing
        expr: rate(voice_errors_total[5m]) > 1
        for: 3m
        labels:
          severity: warning
          service: voice-engine
        annotations:
          summary: 'Voice pipeline errors increasing'
          description: 'Error rate is {{ $value }}/s'
```

### 6.4 GPU Alerts

```yaml
groups:
  - name: psyche-gpu-alerts
    rules:
      - alert: GPUHighUtilization
        expr: gpu_utilization_percent > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'GPU utilization above 90%'
          description: 'GPU {{ $labels.gpu_id }} at {{ $value }}%'

      - alert: GPUCriticalUtilization
        expr: gpu_utilization_percent > 95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'GPU utilization above 95%'
          description: 'GPU {{ $labels.gpu_id }} at {{ $value }}%'

      - alert: GPUMemoryExhaustion
        expr: gpu_memory_used_bytes / gpu_memory_total_bytes > 0.95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'GPU memory nearly exhausted'
          description:
            'GPU {{ $labels.gpu_id }} memory at {{ $value | humanizePercentage
            }}'

      - alert: GPUOverheating
        expr: gpu_temperature_celsius > 85
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'GPU overheating'
          description: 'GPU {{ $labels.gpu_id }} at {{ $value }}°C'

      - alert: GPUQueueBacklog
        expr: sum(gpu_jobs_pending) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'GPU job queue backlog'
          description: '{{ $value }} jobs pending'

      - alert: GPUCircuitBreakerOpen
        expr: gpu_circuit_breaker_state == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'GPU circuit breaker open'
          description: 'Circuit breaker open for {{ $labels.endpoint_id }}'

      - alert: GPUEndpointUnhealthy
        expr: gpu_endpoint_health == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'GPU endpoint unhealthy'
          description: 'Endpoint {{ $labels.endpoint_id }} is unhealthy'
```

---

## 7. Grafana Dashboard Configuration

### 7.1 Dashboard Manifest

```json
{
  "dashboard": {
    "title": "Psyche Platform - Launch Monitoring",
    "uid": "psyche-launch-monitoring",
    "tags": ["psyche", "launch", "production"],
    "timezone": "browser",
    "refresh": "10s",
    "rows": [
      {
        "title": "Overview",
        "panels": [
          "service_health",
          "active_sessions",
          "error_rate",
          "p95_latency"
        ]
      },
      {
        "title": "Avatar Rendering",
        "panels": ["fps", "render_latency", "quality_score", "gpu_memory"]
      },
      {
        "title": "Voice Pipeline",
        "panels": [
          "e2e_latency",
          "latency_breakdown",
          "stt_metrics",
          "tts_metrics"
        ]
      },
      {
        "title": "Error Tracking",
        "panels": [
          "error_rate_by_service",
          "error_distribution",
          "recent_errors"
        ]
      },
      {
        "title": "GPU Utilization",
        "panels": [
          "gpu_pool_status",
          "gpu_utilization",
          "job_queue",
          "circuit_breakers"
        ]
      },
      {
        "title": "Infrastructure",
        "panels": ["database_pool", "cache_hit_rate", "rate_limiter_status"]
      }
    ]
  }
}
```

### 7.2 Key Dashboard Variables

```json
{
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "query",
        "query": "label_values(up, service)",
        "multi": true
      },
      {
        "name": "gpu_id",
        "type": "query",
        "query": "label_values(gpu_utilization_percent, gpu_id)",
        "multi": true
      },
      {
        "name": "session_id",
        "type": "query",
        "query": "label_values(avatar_frames_per_second, session_id)",
        "multi": false
      }
    ]
  }
}
```

---

## 8. AlertManager Configuration

### 8.1 Routing Configuration

```yaml
# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: '${SLACK_WEBHOOK_URL}'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

route:
  group_by: ['alertname', 'service', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'psyche-default'
  routes:
    # Critical alerts -> PagerDuty + Slack
    - match:
        severity: critical
      receiver: 'psyche-critical'
      continue: true

    # Warning alerts -> Slack only
    - match:
        severity: warning
      receiver: 'psyche-warnings'

    # GPU-specific alerts
    - match:
        service: gpu
      receiver: 'psyche-gpu-team'

receivers:
  - name: 'psyche-default'
    slack_configs:
      - channel: '#psyche-alerts'
        send_resolved: true

  - name: 'psyche-critical'
    pagerduty_configs:
      - service_key: '${PAGERDUTY_SERVICE_KEY}'
        severity: critical
    slack_configs:
      - channel: '#psyche-critical'
        send_resolved: true

  - name: 'psyche-warnings'
    slack_configs:
      - channel: '#psyche-warnings'
        send_resolved: true

  - name: 'psyche-gpu-team'
    slack_configs:
      - channel: '#psyche-gpu-ops'
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'service']
```

---

## 9. Launch Readiness Checklist

### 9.1 Monitoring Components

| Component                | Status | Notes                           |
| ------------------------ | ------ | ------------------------------- |
| Prometheus Deployed      | READY  | Scraping all services           |
| Grafana Dashboards       | READY  | 6 dashboard panels configured   |
| Alert Rules              | READY  | 15+ alert rules configured      |
| AlertManager             | READY  | PagerDuty + Slack integration   |
| GPU Metrics Collector    | READY  | All GPU nodes reporting         |
| Voice Pipeline Metrics   | READY  | E2E latency tracking active     |
| Avatar Rendering Metrics | READY  | FPS and quality tracking active |
| Error Rate Tracking      | READY  | Per-service error rates         |

### 9.2 SLO Targets

| SLO                | Target    | Measurement            |
| ------------------ | --------- | ---------------------- |
| Availability       | 99.9%     | Uptime over 30 days    |
| Avatar FPS         | >= 30 FPS | P95 over 1 hour        |
| Voice E2E Latency  | < 500ms   | P95 over 1 hour        |
| Error Rate         | < 0.1%    | Rolling 1 hour average |
| GPU Job Queue Time | < 5s      | P95 over 1 hour        |

---

## 10. Sign-off

| Role           | Name               | Date             | Signature        |
| -------------- | ------------------ | ---------------- | ---------------- |
| SRE Lead       | ********\_******** | ****\_\_\_\_**** | ******\_\_****** |
| Platform Lead  | ********\_******** | ****\_\_\_\_**** | ******\_\_****** |
| Security Lead  | ********\_******** | ****\_\_\_\_**** | ******\_\_****** |
| VP Engineering | ********\_******** | ****\_\_\_\_**** | ******\_\_****** |

---

_Configuration generated: 2026-02-01_ _Platform: Psyche AI Virtual Assistant_
_Document classification: Internal - Operations_
