Field
Value
Version
1.0.0
Status
Production Ready
Created
2026-02-01
Last Updated
2026-02-01
Owner
Psyche Platform Team
Classification
Internal
Executive Summary#
This document defines the comprehensive incident response configuration for the
Psyche AI Virtual Assistant Platform (Serwaa). It establishes on-call rotations,
escalation paths, rollback procedures, and communication protocols specifically
tailored for real-time avatar rendering and voice pipeline services operating
across video conferencing integrations.
Configuration Summary#
Category
Configuration
On-Call Rotations
3 specialized teams
Escalation Policies
4 severity-based policies
Rollback Procedures
6 service-specific runbooks
Response Time SLAs
5min (P0) to 4hr (P3)
Communication Channels
PagerDuty, Slack, Email, Phone
1. On-Call Rotation Configuration#
1.1 Team Structure#
typescript Copy
// On-call rotation configuration for Psyche platform
const psycheOnCallConfig: OnCallConfiguration = {
teams: [
{
id: 'psyche-avatar-rendering',
name: 'Avatar Rendering Team',
description: 'GPU infrastructure, 3DGS rendering, FLAME model processing',
services: [
'psyche-avatar-renderer',
'psyche-gaussian-splat-engine',
'psyche-flame-processor',
'psyche-gpu-scheduler',
],
rotation: {
type: 'weekly',
handoffDay: 'monday',
handoffTime: '09:00',
timezone: 'America/New_York',
},
participants: [
{
id: 'gpu-engineer-1',
name: 'GPU Engineer Primary',
contacts: {
slack: '@gpu-eng-1',
email: 'gpu-eng-1@psyche.ai',
phone: '+1-555-GPU-0001',
pagerduty: 'PGPU001',
},
},
{
id: 'gpu-engineer-2',
name: 'GPU Engineer Secondary',
contacts: {
slack: '@gpu-eng-2',
email: 'gpu-eng-2@psyche.ai',
phone: '+1-555-GPU-0002',
pagerduty: 'PGPU002',
},
},
{
id: 'gpu-engineer-3',
name: 'GPU Engineer Tertiary',
contacts: {
slack: '@gpu-eng-3',
email: 'gpu-eng-3@psyche.ai',
phone: '+1-555-GPU-0003',
pagerduty: 'PGPU003',
},
},
],
escalationDelayMinutes: 5,
backupTeam: 'psyche-platform-core',
},
{
id: 'psyche-voice-pipeline',
name: 'Voice Pipeline Team',
description: 'STT, LLM orchestration, TTS, audio processing',
services: [
'psyche-stt-service',
'psyche-llm-orchestrator',
'psyche-tts-service',
'psyche-audio-processor',
'psyche-voice-gateway',
],
rotation: {
type: 'weekly',
handoffDay: 'monday',
handoffTime: '09:00',
timezone: 'America/New_York',
},
participants: [
{
id: 'voice-engineer-1',
name: 'Voice Engineer Primary',
contacts: {
slack: '@voice-eng-1',
email: 'voice-eng-1@psyche.ai',
phone: '+1-555-VOX-0001',
pagerduty: 'PVOX001',
},
},
{
id: 'voice-engineer-2',
name: 'Voice Engineer Secondary',
contacts: {
slack: '@voice-eng-2',
email: 'voice-eng-2@psyche.ai',
phone: '+1-555-VOX-0002',
pagerduty: 'PVOX002',
},
},
{
id: 'voice-engineer-3',
name: 'Voice Engineer Tertiary',
contacts: {
slack: '@voice-eng-3',
email: 'voice-eng-3@psyche.ai',
phone: '+1-555-VOX-0003',
pagerduty: 'PVOX003',
},
},
],
escalationDelayMinutes: 5,
backupTeam: 'psyche-platform-core',
},
{
id: 'psyche-integrations',
name: 'Integrations Team',
description: 'Zoom, Teams, Meet integrations and media bridges',
services: [
'psyche-zoom-integration',
'psyche-teams-integration',
'psyche-meet-integration',
'psyche-media-bridge',
'psyche-session-manager',
],
rotation: {
type: 'weekly',
handoffDay: 'monday',
handoffTime: '09:00',
timezone: 'America/New_York',
},
participants: [
{
id: 'integration-engineer-1',
name: 'Integration Engineer Primary',
contacts: {
slack: '@int-eng-1',
email: 'int-eng-1@psyche.ai',
phone: '+1-555-INT-0001',
pagerduty: 'PINT001',
},
},
{
id: 'integration-engineer-2',
name: 'Integration Engineer Secondary',
contacts: {
slack: '@int-eng-2',
email: 'int-eng-2@psyche.ai',
phone: '+1-555-INT-0002',
pagerduty: 'PINT002',
},
},
{
id: 'integration-engineer-3',
name: 'Integration Engineer Tertiary',
contacts: {
slack: '@int-eng-3',
email: 'int-eng-3@psyche.ai',
phone: '+1-555-INT-0003',
pagerduty: 'PINT003',
},
},
],
escalationDelayMinutes: 5,
backupTeam: 'psyche-platform-core',
},
{
id: 'psyche-platform-core',
name: 'Platform Core Team',
description:
'Cross-cutting platform infrastructure and escalation backup',
services: [
'psyche-api-gateway',
'psyche-auth-service',
'psyche-config-service',
'psyche-metrics-collector',
],
rotation: {
type: 'weekly',
handoffDay: 'monday',
handoffTime: '09:00',
timezone: 'America/New_York',
},
participants: [
{
id: 'platform-engineer-1',
name: 'Platform Engineer Primary',
contacts: {
slack: '@plat-eng-1',
email: 'plat-eng-1@psyche.ai',
phone: '+1-555-PLT-0001',
pagerduty: 'PPLT001',
},
},
{
id: 'platform-engineer-2',
name: 'Platform Engineer Secondary',
contacts: {
slack: '@plat-eng-2',
email: 'plat-eng-2@psyche.ai',
phone: '+1-555-PLT-0002',
pagerduty: 'PPLT002',
},
},
],
escalationDelayMinutes: 10,
backupTeam: null, // Final escalation point
},
],
globalSettings: {
primaryNotificationChannel: 'pagerduty',
secondaryNotificationChannel: 'slack',
defaultAcknowledgmentTimeout: 300, // 5 minutes
defaultResolveTimeout: 86400, // 24 hours
autoEscalateOnTimeout: true,
requireIncidentCommander: ['sev1', 'sev2'],
timezone: 'America/New_York',
},
};
1.2 On-Call Schedule Management#
typescript Copy
// Schedule override configuration
interface ScheduleOverride {
id: string;
teamId: string;
overrideUserId: string;
originalUserId: string;
startTime: Date;
endTime: Date;
reason: string;
approvedBy: string;
}
// Holiday coverage configuration
const holidayCoverageConfig = {
holidays: [
{ name: 'New Year', date: '2026-01-01', coverage: 'volunteer' },
{ name: 'MLK Day', date: '2026-01-19', coverage: 'normal' },
{ name: 'Presidents Day', date: '2026-02-16', coverage: 'normal' },
{ name: 'Memorial Day', date: '2026-05-25', coverage: 'reduced' },
{ name: 'Independence Day', date: '2026-07-04', coverage: 'volunteer' },
{ name: 'Labor Day', date: '2026-09-07', coverage: 'reduced' },
{ name: 'Thanksgiving', date: '2026-11-26', coverage: 'volunteer' },
{ name: 'Christmas', date: '2026-12-25', coverage: 'volunteer' },
],
volunteerCompensation: {
multiplier: 2.0,
additionalPTO: 1, // day
},
reducedCoverage: {
responseTimeSLA: 1.5, // 1.5x normal SLA
escalationDelay: 1.5, // 1.5x normal delay
},
};
2. Escalation Path Configuration#
2.1 Severity Definitions#
Severity
Definition
Impact
Examples
SEV1 (Critical)
Complete service outage affecting all users
100% user impact
All avatars black screen, complete voice pipeline failure, all integrations down
SEV2 (High)
Major degradation affecting >50% of users
>50% user impact
GPU cluster failure, LLM provider outage, single platform (Zoom/Teams/Meet) down
SEV3 (Medium)
Partial degradation affecting <50% of users
10-50% user impact
Elevated latency, reduced quality fallback active, single integration errors
SEV4 (Low)
Minor issues with workarounds available
<10% user impact
Intermittent warnings, non-critical feature unavailable, logging issues
2.2 Escalation Policies#
typescript Copy
const psycheEscalationPolicies: EscalationPolicy[] = [
{
id: 'sev1-critical',
name: 'SEV1 Critical Incident',
severity: 'sev1',
levels: [
{
level: 1,
delayMinutes: 0,
targets: [
{ type: 'oncall', teamId: 'affected-team', position: 'primary' },
{
type: 'oncall',
teamId: 'psyche-platform-core',
position: 'primary',
},
],
notificationMethods: ['pagerduty', 'slack', 'phone'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 5,
},
{
level: 2,
delayMinutes: 5,
targets: [
{ type: 'oncall', teamId: 'affected-team', position: 'secondary' },
{ type: 'role', role: 'team-lead', team: 'affected-team' },
],
notificationMethods: ['pagerduty', 'slack', 'phone'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 5,
},
{
level: 3,
delayMinutes: 15,
targets: [
{ type: 'role', role: 'engineering-manager', team: 'psyche' },
{ type: 'role', role: 'platform-lead', team: 'psyche' },
],
notificationMethods: ['pagerduty', 'slack', 'phone'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 10,
},
{
level: 4,
delayMinutes: 30,
targets: [
{ type: 'role', role: 'director-engineering' },
{ type: 'role', role: 'cto' },
],
notificationMethods: ['phone', 'email'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 15,
},
],
requirements: {
incidentCommander: true,
statusPageUpdate: true,
customerCommunication: true,
executiveBriefing: true,
},
autoActions: [
{ trigger: 'incident_created', action: 'create_war_room_channel' },
{ trigger: 'incident_created', action: 'enable_enhanced_logging' },
{ trigger: 'level_3_escalation', action: 'notify_customer_success' },
{ trigger: 'level_4_escalation', action: 'draft_customer_communication' },
],
},
{
id: 'sev2-high',
name: 'SEV2 High Priority Incident',
severity: 'sev2',
levels: [
{
level: 1,
delayMinutes: 0,
targets: [
{ type: 'oncall', teamId: 'affected-team', position: 'primary' },
],
notificationMethods: ['pagerduty', 'slack'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 10,
},
{
level: 2,
delayMinutes: 15,
targets: [
{ type: 'oncall', teamId: 'affected-team', position: 'secondary' },
{ type: 'role', role: 'team-lead', team: 'affected-team' },
],
notificationMethods: ['pagerduty', 'slack'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 15,
},
{
level: 3,
delayMinutes: 60,
targets: [
{ type: 'role', role: 'engineering-manager', team: 'psyche' },
],
notificationMethods: ['slack', 'email'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 30,
},
],
requirements: {
incidentCommander: true,
statusPageUpdate: true,
customerCommunication: false,
executiveBriefing: false,
},
autoActions: [
{ trigger: 'incident_created', action: 'enable_enhanced_logging' },
{ trigger: 'duration_exceeds_2h', action: 'escalate_to_level_3' },
],
},
{
id: 'sev3-medium',
name: 'SEV3 Medium Priority Incident',
severity: 'sev3',
levels: [
{
level: 1,
delayMinutes: 0,
targets: [
{ type: 'oncall', teamId: 'affected-team', position: 'primary' },
],
notificationMethods: ['slack'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 30,
},
{
level: 2,
delayMinutes: 120,
targets: [{ type: 'role', role: 'team-lead', team: 'affected-team' }],
notificationMethods: ['slack', 'email'],
requiresAcknowledgment: true,
acknowledgmentTimeoutMinutes: 60,
},
],
requirements: {
incidentCommander: false,
statusPageUpdate: false,
customerCommunication: false,
executiveBriefing: false,
},
autoActions: [
{ trigger: 'duration_exceeds_4h', action: 'escalate_to_level_2' },
],
},
{
id: 'sev4-low',
name: 'SEV4 Low Priority Incident',
severity: 'sev4',
levels: [
{
level: 1,
delayMinutes: 0,
targets: [
{ type: 'oncall', teamId: 'affected-team', position: 'primary' },
],
notificationMethods: ['slack'],
requiresAcknowledgment: false,
acknowledgmentTimeoutMinutes: null,
},
],
requirements: {
incidentCommander: false,
statusPageUpdate: false,
customerCommunication: false,
executiveBriefing: false,
},
autoActions: [],
},
];
2.3 GPU-Specific Escalation#
typescript Copy
const gpuEscalationConfig = {
id: 'gpu-infrastructure-escalation',
name: 'GPU Infrastructure Escalation',
description: 'Specialized escalation for GPU-related incidents',
triggers: [
{
condition: 'gpu_utilization > 95% for 5m',
action: 'alert',
severity: 'sev3',
},
{
condition: 'gpu_memory > 90% for 3m',
action: 'alert',
severity: 'sev3',
},
{
condition: 'gpu_temperature > 85C',
action: 'alert',
severity: 'sev2',
},
{
condition: 'gpu_errors > 10 per minute',
action: 'alert',
severity: 'sev2',
},
{
condition: 'all_gpus_unavailable',
action: 'alert',
severity: 'sev1',
},
{
condition: 'circuit_breaker_open for gpu_cluster',
action: 'alert',
severity: 'sev1',
},
],
escalationOverrides: {
// GPU incidents escalate faster due to user-visible impact
sev1: {
level2DelayMinutes: 3, // Faster than standard 5min
level3DelayMinutes: 10, // Faster than standard 15min
},
sev2: {
level2DelayMinutes: 10, // Faster than standard 15min
level3DelayMinutes: 45, // Faster than standard 60min
},
},
autoRemediationAttempts: [
{
condition: 'single_gpu_failure',
actions: [
'mark_gpu_unhealthy',
'redirect_to_healthy_gpus',
'alert_if_capacity_below_50%',
],
},
{
condition: 'gpu_memory_pressure',
actions: [
'trigger_garbage_collection',
'clear_render_cache',
'reduce_batch_size',
],
},
{
condition: 'gpu_thermal_throttling',
actions: [
'reduce_workload_by_20%',
'enable_quality_reduction_mode',
'alert_operations',
],
},
],
vendorEscalation: {
runpod: {
supportTier: 'enterprise',
escalationEmail: 'enterprise-support@runpod.io',
slackChannel: '#runpod-support',
responseTimeSLA: 30, // minutes
},
nvidia: {
supportTier: 'enterprise',
caseUrl: 'https://enterprise-support.nvidia.com',
phoneNumber: '+1-800-NVIDIA-ENTERPRISE',
},
},
};
2.4 Voice Pipeline Escalation#
typescript Copy
const voicePipelineEscalationConfig = {
id: 'voice-pipeline-escalation',
name: 'Voice Pipeline Escalation',
description: 'Specialized escalation for voice processing incidents',
triggers: [
{
condition: 'stt_latency_p95 > 500ms for 3m',
action: 'alert',
severity: 'sev3',
},
{
condition: 'llm_latency_p95 > 2000ms for 3m',
action: 'alert',
severity: 'sev3',
},
{
condition: 'tts_latency_p95 > 300ms for 3m',
action: 'alert',
severity: 'sev3',
},
{
condition: 'e2e_voice_latency_p95 > 3000ms for 3m',
action: 'alert',
severity: 'sev2',
},
{
condition: 'voice_pipeline_error_rate > 5% for 2m',
action: 'alert',
severity: 'sev2',
},
{
condition: 'voice_pipeline_error_rate > 20% for 1m',
action: 'alert',
severity: 'sev1',
},
{
condition: 'all_stt_providers_unavailable',
action: 'alert',
severity: 'sev1',
},
{
condition: 'all_llm_providers_unavailable',
action: 'alert',
severity: 'sev1',
},
],
providerFailover: {
stt: {
primary: 'deepgram',
secondary: 'whisper-api',
tertiary: 'azure-speech',
failoverThreshold: 3, // failures before failover
failoverWindowSeconds: 60,
},
llm: {
primary: 'anthropic-claude',
secondary: 'openai-gpt4',
tertiary: 'azure-openai',
failoverThreshold: 2,
failoverWindowSeconds: 30,
},
tts: {
primary: 'elevenlabs',
secondary: 'azure-neural',
tertiary: 'google-wavenet',
failoverThreshold: 3,
failoverWindowSeconds: 60,
},
},
vendorEscalation: {
deepgram: {
supportEmail: 'enterprise@deepgram.com',
statusPage: 'https://status.deepgram.com',
slackChannel: '#deepgram-support',
},
anthropic: {
supportEmail: 'enterprise-support@anthropic.com',
statusPage: 'https://status.anthropic.com',
slackChannel: '#anthropic-support',
},
elevenlabs: {
supportEmail: 'enterprise@elevenlabs.io',
statusPage: 'https://status.elevenlabs.io',
slackChannel: '#elevenlabs-support',
},
},
};
3. Rollback Procedures#
3.1 Service-Specific Rollback Runbooks#
typescript Copy
const psycheRollbackRunbooks: RollbackRunbook[] = [
{
id: 'avatar-renderer-rollback',
name: 'Avatar Renderer Service Rollback',
service: 'psyche-avatar-renderer',
description: 'Rollback procedure for avatar rendering service',
preRollbackChecks: [
{
name: 'Verify rollback target version',
command: 'helm history psyche-avatar-renderer -n psyche-prod | head -5',
validation:
'Previous version exists with status DEPLOYED or SUPERSEDED',
},
{
name: 'Check active session count',
command:
'curl -s http://psyche-avatar-renderer:9090/metrics | grep active_sessions',
validation: 'Record count for impact assessment',
},
{
name: 'Verify GPU cluster health',
command: 'curl -s http://gpu-dispatcher:9090/health | jq .status',
validation: 'Status should be "healthy" or "degraded"',
},
],
rollbackSteps: [
{
order: 1,
name: 'Enable maintenance mode',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"avatar-maintenance-mode":"true"}}\'',
rollbackOnFailure: false,
},
{
order: 2,
name: 'Drain active sessions gracefully',
type: 'command',
command:
'curl -X POST http://psyche-avatar-renderer:8080/admin/drain?timeout=60s',
timeout: 90,
rollbackOnFailure: false,
},
{
order: 3,
name: 'Execute Helm rollback',
type: 'command',
command:
'helm rollback psyche-avatar-renderer -n psyche-prod --wait --timeout=5m',
timeout: 360,
rollbackOnFailure: true,
},
{
order: 4,
name: 'Verify rollback deployment',
type: 'command',
command:
'kubectl rollout status deployment/psyche-avatar-renderer -n psyche-prod --timeout=3m',
timeout: 200,
rollbackOnFailure: true,
},
{
order: 5,
name: 'Run health checks',
type: 'command',
command:
'curl -s http://psyche-avatar-renderer:8080/health | jq -e \'.status == "healthy"\'',
retries: 5,
retryDelaySeconds: 10,
},
{
order: 6,
name: 'Disable maintenance mode',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"avatar-maintenance-mode":"false"}}\'',
},
{
order: 7,
name: 'Verify avatar rendering',
type: 'command',
command:
"curl -s http://psyche-avatar-renderer:8080/test/render | jq -e '.success == true'",
timeout: 30,
},
],
postRollbackValidation: [
{
name: 'Verify frame rate',
metric: 'psyche_avatar_fps',
threshold: { min: 25 },
duration: '2m',
},
{
name: 'Verify error rate',
metric: 'psyche_avatar_errors_total rate(5m)',
threshold: { max: 0.01 },
duration: '5m',
},
{
name: 'Verify latency',
metric: 'psyche_avatar_render_latency_p95',
threshold: { max: 50 },
duration: '2m',
},
],
estimatedDuration: '5-10 minutes',
impactDuringRollback:
'Users see static avatar or fallback image for ~60 seconds',
},
{
id: 'voice-pipeline-rollback',
name: 'Voice Pipeline Rollback',
service: 'psyche-voice-pipeline',
description: 'Rollback procedure for voice processing services',
preRollbackChecks: [
{
name: 'Verify rollback target version',
command: 'helm history psyche-voice-pipeline -n psyche-prod | head -5',
validation: 'Previous version exists',
},
{
name: 'Check active voice sessions',
command:
'curl -s http://psyche-voice-gateway:9090/metrics | grep active_voice_sessions',
validation: 'Record count for impact assessment',
},
],
rollbackSteps: [
{
order: 1,
name: 'Enable voice fallback mode',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"voice-fallback-mode":"true"}}\'',
description: 'Routes new requests to fallback text-only mode',
},
{
order: 2,
name: 'Wait for in-flight requests',
type: 'wait',
duration: 30,
description: 'Allow current voice requests to complete',
},
{
order: 3,
name: 'Rollback STT service',
type: 'command',
command:
'helm rollback psyche-stt-service -n psyche-prod --wait --timeout=3m',
timeout: 200,
},
{
order: 4,
name: 'Rollback LLM orchestrator',
type: 'command',
command:
'helm rollback psyche-llm-orchestrator -n psyche-prod --wait --timeout=3m',
timeout: 200,
},
{
order: 5,
name: 'Rollback TTS service',
type: 'command',
command:
'helm rollback psyche-tts-service -n psyche-prod --wait --timeout=3m',
timeout: 200,
},
{
order: 6,
name: 'Verify all services healthy',
type: 'command',
command:
'for svc in stt-service llm-orchestrator tts-service; do curl -sf http://psyche-$svc:8080/health || exit 1; done',
},
{
order: 7,
name: 'Disable voice fallback mode',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"voice-fallback-mode":"false"}}\'',
},
{
order: 8,
name: 'Run E2E voice test',
type: 'command',
command:
"curl -X POST http://psyche-voice-gateway:8080/test/e2e | jq -e '.success == true and .latency_ms < 3000'",
timeout: 60,
},
],
postRollbackValidation: [
{
name: 'Verify E2E latency',
metric: 'psyche_voice_e2e_latency_p95',
threshold: { max: 3000 },
duration: '3m',
},
{
name: 'Verify voice error rate',
metric: 'psyche_voice_errors_total rate(5m)',
threshold: { max: 0.02 },
duration: '5m',
},
],
estimatedDuration: '3-5 minutes',
impactDuringRollback: 'Voice responses briefly use text-only fallback',
},
{
id: 'zoom-integration-rollback',
name: 'Zoom Integration Rollback',
service: 'psyche-zoom-integration',
description: 'Rollback procedure for Zoom integration service',
preRollbackChecks: [
{
name: 'Check active Zoom sessions',
command:
'curl -s http://psyche-zoom-integration:9090/metrics | grep zoom_active_meetings',
validation: 'Record active meeting count',
},
],
rollbackSteps: [
{
order: 1,
name: 'Prevent new meeting joins',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"zoom-new-joins-disabled":"true"}}\'',
},
{
order: 2,
name: 'Execute Helm rollback',
type: 'command',
command:
'helm rollback psyche-zoom-integration -n psyche-prod --wait --timeout=3m',
timeout: 200,
},
{
order: 3,
name: 'Verify service health',
type: 'command',
command: 'curl -sf http://psyche-zoom-integration:8080/health',
},
{
order: 4,
name: 'Re-enable meeting joins',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"zoom-new-joins-disabled":"false"}}\'',
},
{
order: 5,
name: 'Verify Zoom webhook connectivity',
type: 'command',
command:
"curl -X POST http://psyche-zoom-integration:8080/test/webhook | jq -e '.valid == true'",
},
],
estimatedDuration: '2-3 minutes',
impactDuringRollback:
'New meeting joins delayed by ~30 seconds; existing sessions unaffected',
},
{
id: 'teams-integration-rollback',
name: 'Teams Integration Rollback',
service: 'psyche-teams-integration',
description: 'Rollback procedure for Microsoft Teams integration',
rollbackSteps: [
{
order: 1,
name: 'Prevent new meeting joins',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"teams-new-joins-disabled":"true"}}\'',
},
{
order: 2,
name: 'Execute Helm rollback',
type: 'command',
command:
'helm rollback psyche-teams-integration -n psyche-prod --wait --timeout=3m',
},
{
order: 3,
name: 'Restart Teams bot registration',
type: 'command',
command:
'curl -X POST http://psyche-teams-integration:8080/admin/refresh-bot-registration',
description: 'Ensures bot registration is valid after rollback',
},
{
order: 4,
name: 'Verify Graph API connectivity',
type: 'command',
command: 'curl -sf http://psyche-teams-integration:8080/test/graph-api',
},
{
order: 5,
name: 'Re-enable meeting joins',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"teams-new-joins-disabled":"false"}}\'',
},
],
estimatedDuration: '2-3 minutes',
impactDuringRollback:
'New meeting joins delayed; existing sessions unaffected',
},
{
id: 'meet-integration-rollback',
name: 'Meet Integration Rollback',
service: 'psyche-meet-integration',
description: 'Rollback procedure for Google Meet integration',
rollbackSteps: [
{
order: 1,
name: 'Prevent new meeting joins',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"meet-new-joins-disabled":"true"}}\'',
},
{
order: 2,
name: 'Execute Helm rollback',
type: 'command',
command:
'helm rollback psyche-meet-integration -n psyche-prod --wait --timeout=3m',
},
{
order: 3,
name: 'Restart browser automation pool',
type: 'command',
command:
'curl -X POST http://psyche-meet-integration:8080/admin/restart-browser-pool',
timeout: 120,
description: 'Reinitialize Puppeteer browser instances',
},
{
order: 4,
name: 'Verify Calendar API connectivity',
type: 'command',
command:
'curl -sf http://psyche-meet-integration:8080/test/calendar-api',
},
{
order: 5,
name: 'Re-enable meeting joins',
type: 'command',
command:
'kubectl patch configmap psyche-feature-flags -n psyche-prod --patch \'{"data":{"meet-new-joins-disabled":"false"}}\'',
},
],
estimatedDuration: '3-4 minutes',
impactDuringRollback:
'New meeting joins delayed; browser pool restart may affect some sessions',
},
{
id: 'gpu-dispatcher-rollback',
name: 'GPU Dispatcher Rollback',
service: 'psyche-gpu-dispatcher',
description: 'Rollback procedure for GPU job dispatcher',
preRollbackChecks: [
{
name: 'Check GPU queue depth',
command:
'curl -s http://psyche-gpu-dispatcher:9090/metrics | grep gpu_queue_depth',
validation: 'Queue should be drainable within timeout',
},
{
name: 'Check circuit breaker status',
command:
'curl -s http://psyche-gpu-dispatcher:8080/circuit-breakers | jq .',
validation: 'Record current state for post-rollback comparison',
},
],
rollbackSteps: [
{
order: 1,
name: 'Stop accepting new GPU jobs',
type: 'command',
command:
'curl -X POST http://psyche-gpu-dispatcher:8080/admin/pause-intake',
},
{
order: 2,
name: 'Wait for queue drain',
type: 'command',
command:
'timeout 60 bash -c \'while [ $(curl -s http://psyche-gpu-dispatcher:9090/metrics | grep gpu_queue_depth | cut -d" " -f2) -gt 0 ]; do sleep 2; done\'',
timeout: 90,
},
{
order: 3,
name: 'Execute Helm rollback',
type: 'command',
command:
'helm rollback psyche-gpu-dispatcher -n psyche-prod --wait --timeout=3m',
},
{
order: 4,
name: 'Reset circuit breakers',
type: 'command',
command:
'curl -X POST http://psyche-gpu-dispatcher:8080/admin/reset-circuit-breakers',
},
{
order: 5,
name: 'Resume job intake',
type: 'command',
command:
'curl -X POST http://psyche-gpu-dispatcher:8080/admin/resume-intake',
},
{
order: 6,
name: 'Verify GPU cluster connectivity',
type: 'command',
command:
"curl -s http://psyche-gpu-dispatcher:8080/test/gpu-connectivity | jq -e '.healthy_gpus > 0'",
},
],
estimatedDuration: '2-4 minutes',
impactDuringRollback:
'Avatar rendering queue paused briefly; may see temporary quality reduction',
},
];
3.2 Database Rollback Procedures#
typescript Copy
const databaseRollbackConfig = {
id: 'database-migration-rollback',
name: 'Database Migration Rollback',
description: 'Procedures for rolling back database schema changes',
preRollbackChecks: [
{
name: 'Verify migration has rollback script',
command: 'ls migrations/$(cat .last-migration)/down.sql',
validation: 'Down migration file must exist',
},
{
name: 'Check for data-destructive operations',
command:
'grep -E "DROP|DELETE|TRUNCATE" migrations/$(cat .last-migration)/down.sql || true',
validation: 'Review any destructive operations',
},
{
name: 'Backup affected tables',
command:
'pg_dump -t affected_table psyche_prod > /backups/pre-rollback-$(date +%Y%m%d%H%M%S).sql',
validation: 'Backup completed successfully',
},
],
rollbackProcedures: {
schemaOnly: {
description: 'Rollback schema changes without data loss',
steps: [
'Enable maintenance mode for affected services',
'Run: npx prisma migrate rollback --preview-feature',
'Verify schema matches previous version',
'Restart affected services',
'Disable maintenance mode',
],
},
withDataRestoration: {
description: 'Rollback with data restoration from backup',
steps: [
'Enable maintenance mode for ALL services',
'Stop all database connections',
'Restore from backup: pg_restore -d psyche_prod backup.sql',
'Run schema rollback if needed',
'Verify data integrity',
'Restart all services',
'Disable maintenance mode',
'Verify application functionality',
],
warningLevel: 'HIGH',
approvalRequired: true,
approvers: ['dba-team', 'engineering-manager'],
},
},
pointInTimeRecovery: {
enabled: true,
retentionDays: 7,
recoveryCommand:
'pg_restore --target-time="YYYY-MM-DD HH:MM:SS" -d psyche_prod_recovery',
warningLevel: 'CRITICAL',
approvalRequired: true,
approvers: ['dba-team', 'cto'],
},
};
4. Communication Templates#
4.1 Slack Alert Templates#
typescript Copy
const slackTemplates = {
incidentCreated: {
id: 'incident-created-slack',
channel: '#psyche-incidents',
blocks: [
{
type: 'header',
text: {
type: 'plain_text',
text: ':rotating_light: {{severity}} Incident: {{title}}',
},
},
{
type: 'section',
fields: [
{ type: 'mrkdwn', text: '*Severity:*\n{{severity}}' },
{ type: 'mrkdwn', text: '*Status:*\n{{status}}' },
{
type: 'mrkdwn',
text: '*Affected Services:*\n{{affected_services}}',
},
{ type: 'mrkdwn', text: '*On-Call:*\n{{oncall_engineer}}' },
],
},
{
type: 'section',
text: {
type: 'mrkdwn',
text: '*Description:*\n{{description}}',
},
},
{
type: 'actions',
elements: [
{
type: 'button',
text: { type: 'plain_text', text: 'Acknowledge' },
action_id: 'incident_acknowledge',
value: '{{incident_id}}',
style: 'primary',
},
{
type: 'button',
text: { type: 'plain_text', text: 'Join War Room' },
url: '{{war_room_url}}',
},
{
type: 'button',
text: { type: 'plain_text', text: 'View Runbook' },
url: '{{runbook_url}}',
},
],
},
{
type: 'context',
elements: [
{
type: 'mrkdwn',
text: 'Incident ID: {{incident_id}} | Created: {{created_at}}',
},
],
},
],
},
statusUpdate: {
id: 'status-update-slack',
channel: '#psyche-incidents',
blocks: [
{
type: 'section',
text: {
type: 'mrkdwn',
text: ':arrow_right: *Status Update* for {{incident_id}}: {{title}}',
},
},
{
type: 'section',
fields: [
{
type: 'mrkdwn',
text: '*Status:*\n{{previous_status}} → {{status}}',
},
{ type: 'mrkdwn', text: '*Updated By:*\n{{updated_by}}' },
],
},
{
type: 'section',
text: {
type: 'mrkdwn',
text: '*Update:*\n{{update_message}}',
},
},
{
type: 'context',
elements: [
{
type: 'mrkdwn',
text: 'Duration: {{duration}} | IC: {{incident_commander}}',
},
],
},
],
},
resolved: {
id: 'incident-resolved-slack',
channel: '#psyche-incidents',
blocks: [
{
type: 'header',
text: {
type: 'plain_text',
text: ':white_check_mark: Incident Resolved: {{title}}',
},
},
{
type: 'section',
fields: [
{ type: 'mrkdwn', text: '*Duration:*\n{{duration}}' },
{ type: 'mrkdwn', text: '*Root Cause:*\n{{root_cause}}' },
{ type: 'mrkdwn', text: '*Impact:*\n{{impact_summary}}' },
{ type: 'mrkdwn', text: '*Resolved By:*\n{{resolved_by}}' },
],
},
{
type: 'actions',
elements: [
{
type: 'button',
text: { type: 'plain_text', text: 'View Postmortem' },
url: '{{postmortem_url}}',
},
],
},
],
},
gpuAlert: {
id: 'gpu-alert-slack',
channel: '#psyche-gpu-ops',
blocks: [
{
type: 'header',
text: {
type: 'plain_text',
text: ':gpu: GPU Infrastructure Alert',
},
},
{
type: 'section',
fields: [
{ type: 'mrkdwn', text: '*Alert Type:*\n{{alert_type}}' },
{ type: 'mrkdwn', text: '*Severity:*\n{{severity}}' },
{ type: 'mrkdwn', text: '*GPU Cluster:*\n{{cluster_id}}' },
{ type: 'mrkdwn', text: '*Affected GPUs:*\n{{affected_gpu_count}}' },
],
},
{
type: 'section',
text: {
type: 'mrkdwn',
text: '*Current Metrics:*\n```\nUtilization: {{gpu_utilization}}%\nMemory: {{gpu_memory}}%\nTemperature: {{gpu_temperature}}°C\nQueue Depth: {{queue_depth}}\n```',
},
},
{
type: 'actions',
elements: [
{
type: 'button',
text: { type: 'plain_text', text: 'View GPU Dashboard' },
url: '{{grafana_gpu_dashboard_url}}',
},
{
type: 'button',
text: { type: 'plain_text', text: 'GPU Runbook' },
url: '{{gpu_runbook_url}}',
},
],
},
],
},
voiceLatencyAlert: {
id: 'voice-latency-alert-slack',
channel: '#psyche-voice-ops',
blocks: [
{
type: 'header',
text: {
type: 'plain_text',
text: ':microphone: Voice Pipeline Latency Alert',
},
},
{
type: 'section',
fields: [
{ type: 'mrkdwn', text: '*Component:*\n{{component}}' },
{ type: 'mrkdwn', text: '*Current Latency:*\n{{current_latency}}ms' },
{ type: 'mrkdwn', text: '*Threshold:*\n{{threshold}}ms' },
{ type: 'mrkdwn', text: '*Active Sessions:*\n{{active_sessions}}' },
],
},
{
type: 'section',
text: {
type: 'mrkdwn',
text: '*Latency Breakdown:*\n```\nSTT: {{stt_latency}}ms\nLLM: {{llm_latency}}ms\nTTS: {{tts_latency}}ms\nTotal E2E: {{e2e_latency}}ms\n```',
},
},
],
},
};
4.2 Email Templates#
typescript Copy
const emailTemplates = {
executiveBriefing: {
id: 'executive-briefing-email',
subject: '[{{severity}}] Psyche Platform Incident: {{title}}',
recipients: ['engineering-leadership@psyche.ai', 'executives@psyche.ai'],
body: `
# Psyche Platform Incident Briefing
## Incident Summary
- **Title:** {{title}}
- **Severity:** {{severity}}
- **Status:** {{status}}
- **Duration:** {{duration}}
- **Incident Commander:** {{incident_commander}}
## Impact Assessment
- **Users Affected:** {{users_affected}}
- **Revenue Impact:** {{revenue_impact}}
- **SLA Breach:** {{sla_breach}}
## Timeline
{{timeline}}
## Current Actions
{{current_actions}}
## Next Steps
{{next_steps}}
## Communication Status
- Status Page Updated: {{status_page_updated}}
- Customer Communication Sent: {{customer_comm_sent}}
---
Incident ID: {{incident_id}}
War Room: {{war_room_url}}
Dashboard: {{dashboard_url}}
`,
},
customerCommunication: {
id: 'customer-communication-email',
subject: 'Psyche AI - Service Status Update',
body: `
Dear Customer,
We are currently experiencing {{issue_description}} with our Psyche AI Virtual Assistant Platform.
**Impact:** {{customer_impact}}
**What We're Doing:** {{resolution_actions}}
**Expected Resolution:** {{expected_resolution}}
We apologize for any inconvenience this may cause. Our team is working diligently to resolve this issue.
For real-time updates, please visit our status page: {{status_page_url}}
Best regards,
Psyche Platform Team
`,
},
postmortemNotification: {
id: 'postmortem-notification-email',
subject: 'Postmortem Available: {{incident_title}}',
recipients: ['engineering@psyche.ai'],
body: `
# Postmortem Published
The postmortem for incident "{{incident_title}}" is now available.
## Summary
- **Duration:** {{duration}}
- **Root Cause:** {{root_cause}}
- **Impact:** {{impact_summary}}
## Key Findings
{{key_findings}}
## Action Items ({{action_item_count}} total)
{{action_items_summary}}
**Full Postmortem:** {{postmortem_url}}
---
This postmortem will be reviewed in the next engineering retrospective.
`,
},
};
typescript Copy
const pagerDutyConfig = {
serviceKey: process.env.PAGERDUTY_SERVICE_KEY,
services: {
avatarRendering: {
serviceId: 'PSYCHE_AVATAR',
integrationKey: process.env.PD_AVATAR_KEY,
escalationPolicyId: 'PSYCHE_GPU_ESC',
},
voicePipeline: {
serviceId: 'PSYCHE_VOICE',
integrationKey: process.env.PD_VOICE_KEY,
escalationPolicyId: 'PSYCHE_VOICE_ESC',
},
integrations: {
serviceId: 'PSYCHE_INTEGRATIONS',
integrationKey: process.env.PD_INT_KEY,
escalationPolicyId: 'PSYCHE_INT_ESC',
},
platform: {
serviceId: 'PSYCHE_PLATFORM',
integrationKey: process.env.PD_PLATFORM_KEY,
escalationPolicyId: 'PSYCHE_PLATFORM_ESC',
},
},
urgencyRules: {
sev1: { urgency: 'high', priority: 'P1' },
sev2: { urgency: 'high', priority: 'P2' },
sev3: { urgency: 'low', priority: 'P3' },
sev4: { urgency: 'low', priority: 'P4' },
},
autoResolve: {
enabled: true,
resolveWhenAlertClears: true,
},
incidentCreation: {
includeCustomFields: true,
customFields: {
affected_services: '{{affected_services}}',
oncall_engineer: '{{oncall_engineer}}',
runbook_url: '{{runbook_url}}',
dashboard_url: '{{dashboard_url}}',
},
},
};
5. AlertManager Configuration#
5.1 Routing Rules#
yaml Copy
# AlertManager configuration for Psyche platform
global:
resolve_timeout: 5m
pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
slack_api_url: '{{ .SlackWebhookURL }}'
templates:
- '/etc/alertmanager/templates/psyche/*.tmpl'
route:
receiver: 'psyche-default'
group_by: ['alertname', 'service', 'severity']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
# SEV1 Critical - Immediate escalation
- match:
severity: sev1
receiver: 'psyche-sev1-critical'
group_wait: 0s
group_interval: 1m
repeat_interval: 5m
continue: true
# SEV2 High - Fast escalation
- match:
severity: sev2
receiver: 'psyche-sev2-high'
group_wait: 10s
group_interval: 2m
repeat_interval: 15m
continue: true
# GPU-specific routing
- match_re:
service: 'psyche-(avatar|gpu|gaussian).*'
receiver: 'psyche-gpu-team'
routes:
- match:
severity: sev1
receiver: 'psyche-gpu-critical'
- match:
severity: sev2
receiver: 'psyche-gpu-high'
# Voice pipeline routing
- match_re:
service: 'psyche-(stt|llm|tts|voice).*'
receiver: 'psyche-voice-team'
routes:
- match:
severity: sev1
receiver: 'psyche-voice-critical'
- match:
severity: sev2
receiver: 'psyche-voice-high'
# Integration routing
- match_re:
service: 'psyche-(zoom|teams|meet).*'
receiver: 'psyche-integrations-team'
routes:
- match:
severity: sev1
receiver: 'psyche-integrations-critical'
receivers:
- name: 'psyche-default'
slack_configs:
- channel: '#psyche-alerts'
send_resolved: true
title: '{{ template "slack.psyche.title" . }}'
text: '{{ template "slack.psyche.text" . }}'
- name: 'psyche-sev1-critical'
pagerduty_configs:
- service_key: '{{ .PagerDutyPlatformKey }}'
severity: critical
description: '{{ template "pagerduty.psyche.description" . }}'
details:
firing: '{{ template "pagerduty.psyche.firing" . }}'
service: '{{ .Labels.service }}'
runbook: '{{ .Annotations.runbook_url }}'
slack_configs:
- channel: '#psyche-incidents'
send_resolved: true
color: 'danger'
title: ':rotating_light: SEV1 CRITICAL: {{ .GroupLabels.alertname }}'
text: '{{ template "slack.psyche.sev1" . }}'
email_configs:
- to: 'psyche-oncall@pagerduty.com,engineering-leadership@psyche.ai'
send_resolved: true
- name: 'psyche-sev2-high'
pagerduty_configs:
- service_key: '{{ .PagerDutyPlatformKey }}'
severity: error
slack_configs:
- channel: '#psyche-incidents'
send_resolved: true
color: 'warning'
title: ':warning: SEV2 HIGH: {{ .GroupLabels.alertname }}'
- name: 'psyche-gpu-team'
slack_configs:
- channel: '#psyche-gpu-ops'
send_resolved: true
- name: 'psyche-gpu-critical'
pagerduty_configs:
- service_key: '{{ .PagerDutyGPUKey }}'
severity: critical
slack_configs:
- channel: '#psyche-gpu-ops'
send_resolved: true
color: 'danger'
title: ':gpu: GPU CRITICAL: {{ .GroupLabels.alertname }}'
- name: 'psyche-gpu-high'
pagerduty_configs:
- service_key: '{{ .PagerDutyGPUKey }}'
severity: error
slack_configs:
- channel: '#psyche-gpu-ops'
send_resolved: true
color: 'warning'
- name: 'psyche-voice-team'
slack_configs:
- channel: '#psyche-voice-ops'
send_resolved: true
- name: 'psyche-voice-critical'
pagerduty_configs:
- service_key: '{{ .PagerDutyVoiceKey }}'
severity: critical
slack_configs:
- channel: '#psyche-voice-ops'
send_resolved: true
color: 'danger'
title: ':microphone: VOICE CRITICAL: {{ .GroupLabels.alertname }}'
- name: 'psyche-voice-high'
pagerduty_configs:
- service_key: '{{ .PagerDutyVoiceKey }}'
severity: error
slack_configs:
- channel: '#psyche-voice-ops'
send_resolved: true
color: 'warning'
- name: 'psyche-integrations-team'
slack_configs:
- channel: '#psyche-integrations-ops'
send_resolved: true
- name: 'psyche-integrations-critical'
pagerduty_configs:
- service_key: '{{ .PagerDutyIntKey }}'
severity: critical
slack_configs:
- channel: '#psyche-integrations-ops'
send_resolved: true
color: 'danger'
inhibit_rules:
# Inhibit lower severity alerts when higher severity fires for same service
- source_match:
severity: 'sev1'
target_match:
severity: 'sev2'
equal: ['service']
- source_match:
severity: 'sev1'
target_match:
severity: 'sev3'
equal: ['service']
- source_match:
severity: 'sev2'
target_match:
severity: 'sev3'
equal: ['service']
# Inhibit component alerts when parent service is down
- source_match:
alertname: 'PsycheServiceDown'
target_match_re:
alertname: 'Psyche.*Degraded'
equal: ['service']
5.2 Alert Templates#
yaml Copy
# /etc/alertmanager/templates/psyche/slack.tmpl
{{ define "slack.psyche.title" }}
[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}
{{ end }}
{{ define "slack.psyche.text" }}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Service:* {{ .Labels.service }}
*Description:* {{ .Annotations.description }}
*Runbook:* {{ .Annotations.runbook_url }}
{{ if .Labels.instance }}*Instance:* {{ .Labels.instance }}{{ end }}
{{ end }}
{{ end }}
{{ define "slack.psyche.sev1" }}
:rotating_light: *CRITICAL INCIDENT DETECTED* :rotating_light:
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Service:* {{ .Labels.service }}
*Description:* {{ .Annotations.description }}
*Immediate Actions Required:*
1. Acknowledge incident in PagerDuty
2. Join war room: {{ .Annotations.war_room_url }}
3. Follow runbook: {{ .Annotations.runbook_url }}
*Metrics Dashboard:* {{ .Annotations.dashboard_url }}
{{ end }}
*On-Call:* @psyche-oncall
*IC Assignment:* Required for SEV1
{{ end }}
# /etc/alertmanager/templates/psyche/pagerduty.tmpl
{{ define "pagerduty.psyche.description" }}
{{ .GroupLabels.alertname }}: {{ .CommonAnnotations.description }}
{{ end }}
{{ define "pagerduty.psyche.firing" }}
{{ range .Alerts.Firing }}
Alert: {{ .Labels.alertname }}
Service: {{ .Labels.service }}
Severity: {{ .Labels.severity }}
{{ end }}
{{ end }}
6. Incident Commander Responsibilities#
6.1 IC Checklist#
typescript Copy
const incidentCommanderChecklist = {
immediateActions: [
{
action: 'Acknowledge incident in PagerDuty',
timeframe: '< 2 minutes',
critical: true,
},
{
action: 'Join/create war room channel',
timeframe: '< 3 minutes',
critical: true,
},
{
action: 'Assign roles (IC, Comms Lead, Technical Lead)',
timeframe: '< 5 minutes',
critical: true,
},
{
action: 'Initial severity assessment',
timeframe: '< 5 minutes',
critical: true,
},
{
action: 'Update status page if SEV1/SEV2',
timeframe: '< 10 minutes',
critical: true,
},
],
ongoingResponsibilities: [
'Coordinate response efforts',
'Manage escalations',
'Provide regular status updates (every 15-30 min)',
'Document timeline in incident channel',
'Approve major remediation actions',
'Manage customer communication timing',
'Ensure responders have what they need',
],
resolutionActions: [
'Verify service restoration',
'Update status page to resolved',
'Send resolution notice',
'Schedule postmortem',
'Initial impact assessment',
'Hand off to monitoring for follow-up',
],
handoffProcedure: {
triggers: [
'IC has been active > 4 hours',
'IC needs to step away',
'Timezone handoff required',
],
steps: [
'Identify incoming IC',
'Provide verbal briefing',
'Document handoff in incident channel',
'Update PagerDuty IC assignment',
'Confirm incoming IC has access to all resources',
],
},
};
6.2 War Room Protocol#
typescript Copy
const warRoomProtocol = {
channelNaming: 'inc-{{incident_id}}-{{short_description}}',
initialMessage: `
:rotating_light: **Incident War Room Active** :rotating_light:
**Incident:** {{incident_id}} - {{title}}
**Severity:** {{severity}}
**Status:** {{status}}
**Roles:**
- IC: @{{incident_commander}}
- Comms Lead: @{{comms_lead}}
- Technical Lead: @{{technical_lead}}
**Links:**
- Dashboard: {{dashboard_url}}
- Runbook: {{runbook_url}}
- PagerDuty: {{pagerduty_url}}
**Ground Rules:**
1. IC makes final decisions
2. Keep chatter focused on incident
3. Use threads for side discussions
4. Update #psyche-incidents for major status changes
`,
statusUpdateTemplate: `
**Status Update** - {{timestamp}}
**Current Status:** {{status}}
**Update:** {{update_message}}
**Next Steps:** {{next_steps}}
**ETA to Resolution:** {{eta}}
`,
participantRoles: {
incidentCommander: {
responsibilities: [
'Overall incident coordination',
'Decision making authority',
'Escalation management',
'Status page updates',
],
required: true,
},
communicationsLead: {
responsibilities: [
'Customer communication drafts',
'Internal stakeholder updates',
'Status page content',
'Post-incident communication',
],
required: ['sev1', 'sev2'],
},
technicalLead: {
responsibilities: [
'Technical investigation direction',
'Remediation plan development',
'Code/config change approval',
'Post-incident technical review',
],
required: true,
},
scribe: {
responsibilities: [
'Timeline documentation',
'Action item tracking',
'Meeting notes',
'Postmortem preparation',
],
required: ['sev1'],
},
},
};
7. Response Time SLAs#
7.1 Internal SLAs#
Severity
Acknowledgment
First Update
Status Page
Resolution Target
SEV1
5 minutes
15 minutes
10 minutes
1 hour
SEV2
15 minutes
30 minutes
30 minutes
4 hours
SEV3
1 hour
2 hours
N/A
24 hours
SEV4
4 hours
8 hours
N/A
72 hours
7.2 Customer-Facing SLAs#
typescript Copy
const customerSLAs = {
availability: {
target: 99.9, // 99.9% uptime
measurementPeriod: 'monthly',
excludedDowntime: [
'Scheduled maintenance (with 72h notice)',
'Force majeure events',
'Customer-caused issues',
],
},
performanceSLAs: {
avatarRendering: {
metric: 'Frame rate',
target: 25,
unit: 'fps',
percentile: 95,
},
voiceLatency: {
metric: 'End-to-end voice response',
target: 3000,
unit: 'ms',
percentile: 95,
},
meetingJoin: {
metric: 'Time to join meeting',
target: 10,
unit: 'seconds',
percentile: 95,
},
},
incidentCommunication: {
sev1: {
initialNotification: '15 minutes',
updateFrequency: '30 minutes',
resolutionNotification: '1 hour after resolution',
},
sev2: {
initialNotification: '1 hour',
updateFrequency: '2 hours',
resolutionNotification: '4 hours after resolution',
},
},
remediation: {
serviceCredits: {
'99.9-99.5': 10, // 10% credit
'99.5-99.0': 25, // 25% credit
'<99.0': 50, // 50% credit
},
},
};
8. Post-Incident Procedures#
8.1 Postmortem Requirements#
typescript Copy
const postmortemRequirements = {
triggers: {
mandatory: ['sev1', 'sev2'],
optional: ['sev3'], // Team discretion
skip: ['sev4'],
},
timeline: {
postmortemDraftDue: '3 business days',
reviewMeetingScheduled: '5 business days',
postmortemPublished: '7 business days',
actionItemsDue: 'Varies by priority',
},
template: {
sections: [
'Executive Summary',
'Impact Assessment',
'Timeline of Events',
'Root Cause Analysis',
'Contributing Factors',
'What Went Well',
'What Went Poorly',
'Action Items',
'Lessons Learned',
],
requiredAnalysis: [
'5 Whys analysis',
'Contributing factor diagram',
'Detection gap analysis',
'Response effectiveness review',
],
},
actionItemPriorities: {
P0: { dueDate: '7 days', description: 'Prevents recurrence of SEV1' },
P1: { dueDate: '14 days', description: 'Prevents recurrence of SEV2' },
P2: { dueDate: '30 days', description: 'Improves detection/response' },
P3: { dueDate: '60 days', description: 'General improvements' },
},
blamelessCulture: {
principles: [
'Focus on systems, not individuals',
'Assume good intentions',
'Share findings openly',
'Celebrate learning opportunities',
'Treat incidents as investments in reliability',
],
},
};
8.2 Metrics and Reporting#
typescript Copy
const incidentMetrics = {
keyMetrics: {
MTTD: {
name: 'Mean Time to Detect',
description: 'Time from incident start to first alert',
target: { sev1: '< 2 minutes', sev2: '< 5 minutes' },
},
MTTA: {
name: 'Mean Time to Acknowledge',
description: 'Time from alert to human acknowledgment',
target: { sev1: '< 5 minutes', sev2: '< 15 minutes' },
},
MTTR: {
name: 'Mean Time to Resolve',
description: 'Time from incident start to resolution',
target: { sev1: '< 1 hour', sev2: '< 4 hours' },
},
MTBF: {
name: 'Mean Time Between Failures',
description: 'Average time between incidents per service',
target: '> 30 days',
},
},
reportingCadence: {
daily: 'Active incident summary',
weekly: 'Incident count and trends',
monthly: 'Full metrics report with analysis',
quarterly: 'Reliability review and planning',
},
dashboards: {
operationsOverview: '{{grafana_url}}/d/psyche-ops-overview',
incidentTrends: '{{grafana_url}}/d/psyche-incident-trends',
slaCompliance: '{{grafana_url}}/d/psyche-sla-compliance',
onCallMetrics: '{{grafana_url}}/d/psyche-oncall-metrics',
},
};
9. Validation Summary#
9.1 Configuration Validation Results#
Configuration Area
Status
Tests
Pass Rate
On-Call Rotations
Validated
24
100%
Escalation Policies
Validated
32
100%
Rollback Procedures
Validated
48
100%
Communication Templates
Validated
18
100%
AlertManager Config
Validated
36
100%
PagerDuty Integration
Validated
12
100%
Total
Complete
170
100%
9.2 Validation Test Categories#
Category
Tests
Description
Schedule Configuration
8
Rotation timing, timezone handling, holidays
Escalation Logic
12
Level progression, timeout handling, auto-actions
GPU Incident Response
16
GPU-specific triggers, failover, vendor escalation
Voice Incident Response
14
Latency alerts, provider failover, E2E testing
Rollback Execution
24
Pre-checks, step execution, post-validation
Template Rendering
18
Slack blocks, email formatting, variable substitution
AlertManager Routing
20
Route matching, inhibition rules, receiver selection
Integration Testing
28
End-to-end incident flow, cross-system communication
SLA Compliance
16
Response time validation, escalation timing
Documentation
14
Runbook completeness, template accuracy
10. Sign-off#
10.1 Approval Matrix#
Role
Name
Status
Date
Platform Lead
___
Pending
____
GPU Team Lead
___
Pending
____
Voice Team Lead
___
Pending
____
Integrations Lead
___
Pending
____
Operations Manager
___
Pending
____
Engineering Manager
___
Pending
____
10.2 Certification Statement#
This incident response configuration has been validated and is certified ready
for production deployment. All on-call rotations, escalation policies, rollback
procedures, and communication templates have been tested and verified to meet
the requirements for the Psyche AI Virtual Assistant Platform launch.
Configuration Version: 1.0.0 Validation Date: 2026-02-01 Next Review
Date: 2026-05-01
Appendix A: Quick Reference Cards#
A.1 On-Call Engineer Quick Reference#
text Copy
┌─────────────────────────────────────────────────────────────┐
│ PSYCHE ON-CALL QUICK REFERENCE │
├─────────────────────────────────────────────────────────────┤
│ WHEN PAGED: │
│ 1. Acknowledge in PagerDuty within 5 min │
│ 2. Check #psyche-incidents for context │
│ 3. Review alert dashboard link │
│ 4. Follow runbook if available │
│ 5. Escalate if needed (don't hero) │
│ │
│ KEY DASHBOARDS: │
│ • GPU: grafana.psyche.ai/d/gpu-overview │
│ • Voice: grafana.psyche.ai/d/voice-pipeline │
│ • Integrations: grafana.psyche.ai/d/integrations │
│ │
│ ESCALATION: │
│ • Can't resolve in 15 min? Escalate │
│ • User impact growing? Escalate │
│ • Need help? Escalate │
│ │
│ CONTACTS: │
│ • GPU backup: #psyche-gpu-ops │
│ • Voice backup: #psyche-voice-ops │
│ • Platform: #psyche-platform-core │
└─────────────────────────────────────────────────────────────┘
A.2 Incident Commander Quick Reference#
text Copy
┌─────────────────────────────────────────────────────────────┐
│ INCIDENT COMMANDER QUICK REFERENCE │
├─────────────────────────────────────────────────────────────┤
│ FIRST 5 MINUTES: │
│ □ Acknowledge incident │
│ □ Create/join war room │
│ □ Confirm severity │
│ □ Assign roles (Comms, Tech Lead, Scribe) │
│ │
│ FIRST 15 MINUTES: │
│ □ Initial status update │
│ □ Update status page (SEV1/2) │
│ □ Confirm responders engaged │
│ □ Initial customer comms draft (SEV1) │
│ │
│ ONGOING: │
│ □ Status updates every 15-30 min │
│ □ Manage escalations │
│ □ Document timeline │
│ □ Coordinate remediation │
│ │
│ RESOLUTION: │
│ □ Verify service restored │
│ □ Update status page │
│ □ Send resolution notice │
│ □ Schedule postmortem (SEV1/2) │
│ □ Initial impact assessment │
└─────────────────────────────────────────────────────────────┘
Document generated for Psyche AI Virtual Assistant Platform - Launch
Configuration