Psyche Domain — Architecture — Psyche domain

Psyche (Serwaa) - Hyper-Realistic AI Virtual Assistant Platform

Psyche is the domain responsible for photorealistic digital-human virtual assistants that join real video conference meetings, hold real-time conversations, and perform computer tasks on behalf of participants. The core engineering challenge is achieving human indistinguishability in real time: rendering a convincing 3D human face, synthesizing natural voice, tracking participant emotion, and generating intelligent responses — all within a sub-500ms end-to-end latency budget, inside a live Zoom, Microsoft Teams, Google Meet, or Webex call.

The architecture decomposes this into six loosely coupled real-time engines (Avatar, Voice, Behavior, Perception, Computer-Use, Conferencing), each running as an independent Python microservice. An Orchestrator service coordinates session lifecycle and routes data across engines; a Persona Service manages configurable identity profiles; and a Knowledge Base service supplies domain-specific facts via Retrieval-Augmented Generation. All engines communicate via gRPC for binary streams and Redis pub/sub for state broadcasts, ensuring that no single slow component blocks the others.

A key design choice is graceful degradation: if the local GPU is overloaded, avatar rendering falls back to Tavus cloud rendering; if the primary TTS provider is unavailable, the voice pipeline switches to the next in a failover chain; the conversation never drops. This document describes the full system topology, data flows, library organization, ML pipelines, and integration architecture.

1. Design Principles#

Human Indistinguishability — All rendering, voice, and behavioral systems are tuned to avoid the uncanny valley. Quality wins over speed.
Real-Time First — Sub-500ms end-to-end latency is a hard requirement. Every component budget is tracked.
Modular Pipelines — Avatar, voice, behavior, and perception are independent services that communicate via gRPC and Redis pub/sub, not monolithic code.
Privacy by Default — Session audio and video are not retained beyond the session. Memory is opt-in. Memory deletion is always honored.
Graceful Degradation — If the avatar engine is overloaded, fall back to Tavus cloud rendering. If ElevenLabs is down, fall back to XTTS. The conversation never drops.
Provider Agnosticism — Voice providers, STT providers, LLMs, and conferencing platforms are all pluggable. Psyche does not lock in to any single vendor.

2. Technology Stack#

The table below summarizes every technology layer. Most backend services are Python FastAPI applications; TypeScript libraries provide client-facing SDKs, observability tooling, and cross-domain bridges. GPU-accelerated components (avatar rendering, voice synthesis, perception) run on CUDA 12.1+ with TensorRT for optimized inference.

Component	Technology
Backend Services	Python 3.11+, FastAPI, uvicorn, asyncio
TypeScript Libs	TypeScript 5+, Node.js 22+, Vitest
Admin Dashboard	TypeScript, Next.js 14, Tailwind CSS
Avatar Rendering	3D Gaussian Splatting, FLAME model, CUDA 12.1+, TensorRT
Voice Synthesis	XTTS v2, ElevenLabs, Cartesia, OpenAI TTS, Deepgram Aura
Speech Recognition	Deepgram Nova-2 (primary), OpenAI Whisper (local/fallback)
LLM	Anthropic Claude 3.5 Sonnet (primary), GPT-4, Gemini, Ollama
Embeddings	Voyage AI (primary), Cohere, OpenAI (fallbacks)
Database	PostgreSQL 16 with pgvector, SQLAlchemy ORM
Vector Store	Qdrant
Cache	Redis 7
Object Storage	MinIO (S3-compatible)
Conferencing	Zoom SDK, Microsoft Teams (Graph API + Bot Framework), WebRTC, Webex SDK
Cloud Avatars	Tavus Phoenix (optional)
Build System	Nx + pnpm (TypeScript), Poetry (Python)
Code Quality	ESLint + Prettier (TS), Ruff + mypy (Python)
Infrastructure	Docker, Kubernetes, NVIDIA GPU nodes

3. High-Level System Diagram#

The diagram below shows how external clients connect through the API Gateway to the three layers of the system: management services (Orchestrator, Persona, Knowledge Base), the real-time processing engines, and the underlying data stores. GPU memory allocations are shown in brackets for the three GPU-bound engines.

text

┌─────────────────────────────────────────────────────────────────────────────┐
│                             EXTERNAL CLIENTS                                 │
│                                                                              │
│  Web App  |  Mobile App  |  REST API  |  Zoom / Teams / Meet / Webex         │
└──────────────────────────────┬───────────────────────────────────────────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │   API Gateway :8000   │
                    │  Auth | Rate Limit    │
                    │  Routing | Sessions   │
                    └──────────┬───────────┘
                               │
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────┐
│   ORCHESTRATOR   │ │  PERSONA SERVICE │ │    KNOWLEDGE BASE    │
│     :8007        │ │     :8009        │ │       :8008          │
│                  │ │                  │ │                      │
│  Session State   │ │  Identity Mgmt   │ │  Document Ingestion  │
│  Pipeline Coord  │ │  Personality     │ │  Vector Embeddings   │
│  Resource Alloc  │ │  Voice Config    │ │  Semantic Search     │
└────────┬─────────┘ └──────────────────┘ └──────────────────────┘
         │
         │  Session Coordination
         ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          REAL-TIME PROCESSING LAYER                          │
│                                                                              │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐              │
│  │  AVATAR ENGINE  │  │  VOICE ENGINE   │  │ BEHAVIOR ENGINE │              │
│  │   Port 8001     │  │   Port 8002     │  │   Port 8003     │              │
│  │   [GPU: 8 GB]   │  │   [GPU: 6 GB]   │  │   [CPU only]    │              │
│  │                 │  │                 │  │                 │              │
│  │  3DGS Render    │  │  XTTS / EL TTS  │  │  FACS Exprs.    │              │
│  │  FLAME Model    │  │  Whisper STT    │  │  Micro-Exprs.   │              │
│  │  Lip Sync       │  │  Voice Cloning  │  │  Gaze Control   │              │
│  │  TAA            │  │  Viseme Gen     │  │  Head Movement  │              │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘              │
│           │                    │                    │                        │
│  ┌────────┴───────┐  ┌─────────┴──────┐  ┌─────────┴──────────┐             │
│  │ PERCEPTION ENG │  │  CONFERENCING  │  │   COMPUTER USE     │             │
│  │   Port 8004    │  │   Port 8006    │  │    Port 8005       │             │
│  │   [GPU: 4 GB]  │  │                │  │                    │             │
│  │                │  │  Zoom SDK      │  │  Headless Chrome   │             │
│  │  Face Detect   │  │  Teams SDK     │  │  Screen Analysis   │             │
│  │  Emotion Recog │  │  Meet WebRTC   │  │  Sandboxed Exec    │             │
│  │  Gaze Track    │  │  Webex SDK     │  │  Tool Framework    │             │
│  └────────────────┘  └────────────────┘  └────────────────────┘             │
└─────────────────────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             DATA LAYER                                       │
│                                                                              │
│  ┌────────────┐  ┌───────────┐  ┌────────────┐  ┌────────────────────────┐  │
│  │ PostgreSQL │  │   Redis   │  │   Qdrant   │  │    MinIO (S3)          │  │
│  │ (pgvector) │  │           │  │            │  │                        │  │
│  │ Personas   │  │ Sessions  │  │ Knowledge  │  │ serwaa-models          │  │
│  │ Sessions   │  │ Cache     │  │ Memories   │  │ serwaa-avatars         │  │
│  │ Knowledge  │  │ Pub/Sub   │  │ Voices     │  │ serwaa-voices          │  │
│  │ Audit      │  │ Queues    │  │ Convs.     │  │ serwaa-knowledge       │  │
│  └────────────┘  └───────────┘  └────────────┘  └────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────┘

4. Library Organization#

Psyche ships 135 library directories under libs/psyche/ (predominantly TypeScript) plus 16 Python services under services/psyche/. Most libs/psyche/ libraries are TypeScript packages (package.json + tsup.config.ts); a small number — including auth, cache, and database — are Python packages (pyproject.toml).

The libraries are grouped below by functional area. Each group corresponds to a dedicated service or subsystem in the architecture diagram above.

text

libs/psyche/
│
├── common/            # Shared types, utilities, constants
├── storage/           # S3-compatible file storage
├── logging/           # Structured logging
├── tracing/           # Distributed tracing
├── messaging/         # Inter-service messaging
│
├── Avatar System
│   ├── avatar-core/         # Core rendering pipeline
│   ├── avatar-cache/        # Model caching and preloading
│   ├── avatar-expressions/  # Facial expression system
│   ├── avatar-lipsync/      # Lip synchronization
│   ├── avatar-quality/      # Quality metrics
│   ├── avatar-taa/          # Temporal anti-aliasing
│   ├── avatar-training/     # Training pipeline
│   └── avatar-isis-integration/ # Cross-domain avatar generation
│
├── Voice and Speech
│   ├── voice-synthesis/   # TTS synthesis
│   ├── voice-streaming/   # Real-time audio streaming
│   ├── voice-consistency/ # Quality monitoring
│   ├── speech-recognition/# STT
│   ├── viseme-generator/  # Visual phoneme generation
│   └── noise-handling/    # Audio preprocessing
│
├── Behavior and Expression
│   ├── emotion-engine/            # Emotional response generation
│   ├── emotion-recognition/       # User emotion detection
│   ├── micro-expressions/         # Subtle facial expressions
│   ├── facs-expressions/          # FACS control system
│   ├── gaze-control/              # Eye gaze direction
│   ├── gaze-estimation/           # Participant gaze tracking
│   ├── head-movement/             # Head motion generation
│   ├── head-pose-estimation/      # Participant head pose detection
│   ├── posture-system/            # Body posture management
│   ├── gesture-system/            # Hand/body gestures
│   ├── behavior-coordinator/      # Multi-channel orchestration
│   ├── behavior-anomaly-detector/ # Anomaly detection
│   └── uncanny-valley/            # Uncanny valley mitigation
│
├── Perception
│   ├── face-detection/      # Video stream face detection
│   ├── face-analysis/       # Facial feature analysis
│   ├── engagement-detector/ # Engagement measurement
│   ├── attention-tracker/   # Participant attention
│   └── participation-tracker/ # Meeting contribution analysis
│
├── Memory System
│   ├── memory-core/          # Core memory manager
│   ├── memory-working/       # Short-term working memory
│   ├── memory-in-context/    # In-context memory injection
│   ├── memory-embeddings/    # Memory vector embedding
│   ├── memory-retrieval/     # Retrieval and recall
│   ├── memory-archival/      # Long-term archival
│   ├── memory-consolidation/ # Memory consolidation
│   ├── memory-persistence/   # Database persistence
│   └── memory-tools/         # Memory tool implementations
│
├── Knowledge and Intelligence
│   ├── knowledge-context/   # Knowledge context management
│   ├── knowledge-ingestion/ # Document ingestion pipeline
│   ├── knowledge-retrieval/ # RAG retrieval
│   └── dialogue-manager/    # Conversation flow management
│
├── Computer Use
│   ├── computer-use-core/  # Core orchestration
│   ├── browser-automation/ # Headless browser control
│   ├── screen-analysis/    # Screen content understanding
│   ├── sandbox/            # Secure execution sandbox
│   ├── tool-registry/      # Tool registration
│   ├── tool-execution/     # Tool execution engine
│   └── action-safety/      # Action safety validation
│
├── Video Conferencing
│   ├── conferencing-core/   # Core conferencing abstraction
│   ├── zoom-integration/    # Zoom SDK integration
│   ├── teams-integration/   # Microsoft Teams integration
│   ├── meet-integration/    # Google Meet integration
│   ├── webex-integration/   # Webex integration
│   ├── participant-manager/ # Participant management
│   └── state-sync/          # State synchronization
│
├── Tavus Integration (optional cloud rendering)
│   └── tavus-bridge, tavus-client, tavus-conversation,
│       tavus-expression, tavus-hybrid, tavus-knowledge,
│       tavus-llm, tavus-memories, tavus-objectives,
│       tavus-perception, tavus-persona-manager, tavus-phoenix,
│       tavus-pipecat, tavus-pipeline, tavus-replica-manager,
│       tavus-tools, tavus-tts, tavus-turntaking
│
├── Observability and Quality
│   ├── latency-analysis/     # End-to-end latency measurement
│   ├── slo-alerting/         # SLO monitoring
│   ├── nps/                  # Net Promoter Score
│   ├── satisfaction/         # User satisfaction
│   ├── human-evaluation/     # Human evaluation framework
│   └── k8s-security/         # Kubernetes security policies
│
└── Cross-Domain
    ├── sophia-integration/         # Sophia knowledge search bridge
    ├── sophia-search-integration/  # Sophia semantic search bridge
    └── recall-integration/         # Recall.ai bridge

Note on Python libraries inside libs/psyche/: auth (authentication/authorization), database (schema, Prisma, migrations), and cache (Redis caching) are Python packages (pyproject.toml, Python sources only) that live inside libs/psyche/ alongside the TypeScript libraries — the libs/psyche/ tree is therefore not exclusively TypeScript. They are distinct from the deployable Python services under services/psyche/ listed in Section 5.

5. Service Architecture#

Python Backend Microservices#

All backend services are Python FastAPI applications under services/psyche/. There are 16 service directories; the project.json name is in parentheses. Three of these (Behavior Engine, Security, Error Handling) are currently service shells — the directory and pyproject.toml exist but src/ has not yet been populated with logic.

Service (project name)	Responsibilities
API Gateway (`psyche-api-gateway`)	REST + WebSocket gateway, rate limiting, request routing
Orchestrator (`psyche-orchestrator`)	Session state machine, pipeline routing, resource allocation
Persona Service (`psyche-persona-service`)	Persona identity/personality/expertise/memory/constraints
Knowledge Base (`psyche-knowledge-base`)	Document ingestion, chunking, embedding, RAG, retrieval
Avatar Engine (`psyche-avatar-engine`)	3DGS / NeRF rendering, FLAME model, lip sync, TAA
Voice Engine (`psyche-voice-engine`)	TTS, STT, diarization, visemes, turn-taking, captions
Behavior Engine (`psyche-behavior-engine`)	Service shell — `src/` is currently empty
Perception Engine (`psyche-perception-engine`)	Face/emotion/intent/screen perception
Computer Use (`psyche-computer-use`)	Browser/desktop automation agent, vision, sandbox
Conferencing (`psyche-conferencing`)	Zoom/Teams/Meet/Webex/WebRTC adapters, audio/video routing
Video Conferencing (`psyche-video-conferencing`)	Conferencing service shell
Learning System (`psyche-learning-system`)	Continuous adaptation, persona improvement from feedback
Tool Framework (`psyche-tool-framework`)	Tool registry, router, MCP server
Tavus Integration (`psyche-tavus-integration`)	Tavus cloud-rendering integration
Security (`psyche-security`)	Security service shell
Error Handling (`psyche-error-handling`)	Error-handling service shell

The deployable services above are distinct from the Python libraries (auth, cache, database) that live under libs/psyche/. There is no dedicated standalone "Conferencing Gateway" service; conferencing logic is in the psyche-conferencing package (whose Python module is video_conferencing).

6. Session Lifecycle Flow#

When a client requests a session, the API Gateway delegates to the Orchestrator, which allocates resources and coordinates the startup of the Avatar, Voice, and Conferencing engines in parallel before signaling readiness. The sequence below shows the request path from session creation through the active state.

text

[Client]              [API Gateway]          [Orchestrator]
    │                       │                      │
    │  POST /sessions        │                      │
    │ ─────────────────────>│                      │
    │                       │  Create record        │
    │                       │ ────────────────────>│
    │                       │  Allocate resources   │
    │                       │<─────────────────────│
    │  session_id, PENDING  │                      │
    │<──────────────────────│                      │
    │                       │                      │
    │  POST /sessions/{id}/join                    │
    │ ─────────────────────>│                      │
    │                       │                      │
    │                       │  [Conferencing :8006] │
    │                       │  Connect to meeting   │
    │                       │                      │
    │                       │  [Avatar :8001]       │
    │                       │  Load 3DGS model      │
    │                       │                      │
    │                       │  [Voice :8002]        │
    │                       │  Warm up TTS          │
    │                       │                      │
    │  status=ACTIVE        │                      │
    │<──────────────────────│                      │
    │                       │                      │
    ▼                       ▼                      ▼
         ════════ ACTIVE SESSION PROCESSING ════════
                   (Real-Time Processing Loop)

7. Real-Time Processing Loop#

Once a session is active, every frame of the meeting goes through a pipelined chain of operations. Incoming audio and video are first analyzed by the Perception Engine (detecting emotion and gaze), then speech-to-text transcription drives the LLM and TTS, and finally the Behavior Engine drives the Avatar Engine to render the response video. The total budget for this cycle is under 500ms.

text

[Video Conference]                                    [AI Avatar Output]
      │                                                       ▲
      │ User Audio + Video (30 FPS)                           │
      ▼                                                       │
┌──────────────┐    ┌──────────────┐    ┌─────────────────┐  │
│ CONFERENCING │───>│  PERCEPTION  │───>│   ORCHESTRATOR  │  │
│   Gateway    │    │    Engine    │    │                 │  │
│              │    │              │    │  Route to LLM   │  │
│  Audio/Video │    │  Face Detect │    │  Manage context │  │
│  30 FPS      │    │  Emotion     │    │  Tool execution │  │
└──────────────┘    │  Head Pose   │    └────────┬────────┘  │
                    │  Gaze        │             │           │
                    └──────────────┘             │           │
                                                 ▼           │
                                    ┌────────────────────┐   │
                                    │    VOICE ENGINE    │   │
                                    │  STT (Deepgram)    │   │
                                    │  Claude LLM        │   │
                                    │  TTS (ElevenLabs)  │   │
                                    │  Viseme Generation │   │
                                    └────────┬───────────┘   │
                                             │               │
                          ┌──────────────────┼───────────────┤
                          │                  │               │
                          ▼                  ▼               │
                   ┌────────────┐    ┌────────────────┐      │
                   │  BEHAVIOR  │    │  AVATAR ENGINE │──────┘
                   │   Engine   │───>│                │
                   │  FACS      │    │  3DGS Render   │
                   │  Gestures  │    │  FLAME Model   │
                   │  Gaze      │    │  1080p @ 30 FPS│
                   └────────────┘    └────────────────┘

TIMING BUDGET:
  Perception:       < 20 ms
  Voice STT:        < 100 ms
  LLM Response:     < 200 ms
  Voice TTS:        < 100 ms
  Behavior Compute: < 10 ms
  Avatar Render:    < 33 ms
  TOTAL:            < 500 ms

8. Data Architecture#

Psyche uses three distinct persistence mechanisms, each chosen for a specific access pattern. PostgreSQL holds structured relational data (personas, sessions, audit logs). Qdrant provides approximate nearest-neighbor search over high-dimensional vectors for knowledge and memory retrieval. MinIO holds large binary objects — trained avatar models, cloned voice files, and uploaded documents. Redis sits in front of all three as a multi-TTL caching layer, keeping the hot paths well under 5ms.

text

                        APPLICATION LAYER
  ┌─────────────┬─────────────┬───────────────┬───────────────────────┐
  │ API Gateway │ Orchestrator│  Avatar/Voice │ Other Microservices   │
  └──────┬──────┴──────┬──────┴───────┬───────┴────────────┬──────────┘
         │             │              │                    │
         ▼             ▼              ▼                    ▼
                  CACHING LAYER
  ┌────────────────────────────────────────────────────────────────────┐
  │  L1: Application-Level (< 1 ms)   │  L2: Redis Cluster (< 5 ms)  │
  │  Hot config data                   │  Session state (TTL: 1 hr)   │
  │  Recent embeddings                 │  Persona config (TTL: 30min) │
  │  Pinned model weights              │  Avatar state (TTL: 1 min)   │
  │                                    │  Voice state (TTL: 30 sec)   │
  └────────────────────────────────────────────────────────────────────┘
         │             │              │                    │
         ▼             ▼              ▼                    ▼
                  PERSISTENCE LAYER
  ┌─────────────┐  ┌───────────────┐  ┌────────────────────────────┐
  │  PostgreSQL │  │    Qdrant     │  │       MinIO (S3)           │
  │  (pgvector) │  │ Vector search │  │  Object storage            │
  │             │  │ for knowledge │  │  (avatar models, voices,   │
  │  Target     │  │ and memory    │  │   documents)               │
  │  store for  │  │ embeddings    │  │                            │
  │  domain     │  │               │  │                            │
  │  data       │  │               │  │                            │
  └─────────────┘  └───────────────┘  └────────────────────────────┘

Implementation status. This is the intended persistence design. As built, the API Gateway keeps personas, sessions, knowledge documents, tools, and webhooks in per-process in-memory dictionaries (every router file is labelled "In-memory storage for demo purposes"), and its lifespan handler leaves database and Redis initialization commented out. The only database DDL that exists is the idempotent bootstrap in api-gateway/.../db/migrate.py, which enables extensions (including pgvector) and creates the persona, knowledge, sessions, analytics, and audit schemas plus a single audit.change_log table — no domain tables. SQLAlchemy/asyncpg are declared dependencies but no connections are opened at runtime.

9. ML Pipelines#

Psyche has three offline training pipelines — avatar generation, voice cloning, and knowledge ingestion — each producing artifacts that are stored in MinIO and consumed by the real-time rendering and retrieval paths at session time.

Avatar Training Pipeline#

An avatar is created from a short video of the target person. The pipeline extracts face geometry, fits the FLAME parametric face model to capture expression controls, and then runs a 3D Gaussian Splatting optimization over thousands of iterations. The resulting model is TensorRT-optimized for real-time inference.

text

Input: 30–60s training video (1080p+)
          │
          v
    Face Extraction
    (MTCNN detector, landmark tracking)
          │
          v
    FLAME Model Fitting
    (50 expression blend shapes)
          │
          v
    3DGS Training
    (~30,000 iterations, ~1–4 hours)
          │
          v
    Expression Blend Shape Learning
    (lip sync, emotion blends)
          │
          v
    TensorRT Optimization
    (optimized for real-time rendering)
          │
          v
    Model Storage (MinIO: serwaa-models)

Voice Cloning Pipeline#

A cloned voice is derived from audio samples of the target person. The pipeline extracts a speaker embedding that captures the vocal identity, then fine-tunes the synthesis model to reproduce that speaker's characteristics.

text

Input: 1–3 min audio samples (WAV/MP3, 44.1 kHz+, SNR > 20 dB)
          │
          v
    Audio Preprocessing
    (noise removal, normalization)
          │
          v
    Speaker Embedding Extraction
    (SpeakerNet or XTTS encoder)
          │
          v
    Voice Model Training / Fine-tuning
          │
          v
    Voice Model Storage (MinIO: serwaa-voices)

Knowledge Ingestion Pipeline#

Documents uploaded to the knowledge base are parsed, split into overlapping semantic chunks, embedded into high-dimensional vectors by Voyage AI, and indexed into Qdrant. At inference time, a semantic nearest-neighbor search retrieves the most relevant chunks for RAG injection.

text

Input: PDF / DOCX / TXT / HTML / URL
          │
          v
    Document Parser (format-specific)
          │
          v
    Text Extraction + Metadata
          │
          v
    Semantic Chunking
    (512 tokens, 64 overlap)
          │
          v
    Embedding Generation (Voyage AI)
          │
          v
    Qdrant Indexing
    (collection: psyche_knowledge)
          │
          v
    PostgreSQL Metadata Record

10. Communication Patterns#

Services communicate using three mechanisms, each chosen for the characteristics of the data being exchanged. Synchronous REST calls handle configuration and management operations. Redis pub/sub broadcasts lightweight state-change events to all interested subscribers without coupling publishers to receivers. gRPC streaming handles high-bandwidth continuous data such as audio and video frames.

Service-to-Service REST (Python → Python)#

Standard HTTP requests using httpx.AsyncClient for configuration queries, persona lookups, and other request-response operations.

python

async def get_persona_config(persona_id: str) -> PersonaConfig:
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"{PERSONA_SERVICE_URL}/api/v1/personas/{persona_id}"
        )
        return PersonaConfig(**response.json())

Redis Pub/Sub (real-time event broadcasting)#

State changes are published on named channels and consumed by any number of subscribers. For example, the Avatar Engine publishes its current state and the Behavior Engine subscribes to drive expression synchronization.

python

# Publisher (Avatar Engine)
await redis.publish(f"avatar:{session_id}:state", state.model_dump_json())

# Subscriber (Behavior Engine)
pubsub = redis.pubsub()
await pubsub.subscribe(f"avatar:{session_id}:state")
async for message in pubsub.listen():
    yield AvatarState.model_validate_json(message["data"])

gRPC Streaming (audio/video streams)#

Binary audio and video streams use bidirectional gRPC streaming for minimum latency. The service definitions below show both the video frame and audio chunk streams exposed by the Avatar Engine.

protobuf

service AvatarService {
  rpc StreamFrames(stream RenderRequest) returns (stream VideoFrame);
  rpc StreamAudio(stream AudioChunk) returns (stream AudioChunk);
}

11. Security Architecture#

Authentication#

JWT tokens — Short-lived (1 hr) signed tokens issued by API Gateway
API keys — HMAC-signed keys with organization scope and expiry
Webhook HMAC — All outbound webhooks include HMAC-SHA256 signature header

Data Protection#

Session audio and video are processed in-memory only and not persisted
Conversation transcripts are encrypted at rest (AES-256-GCM)
Memory records are encrypted; deletion is permanent
MinIO objects use server-side encryption

Computer Use Sandbox#

The computer-use browser runs in a tightly constrained environment to prevent accidental or malicious access to host resources. Key restrictions include:

Isolated headless Chromium with no access to host filesystem
Network access restricted to explicit allowlist per tool
Maximum 50 actions per task (configurable)
High-risk actions require human confirmation before execution

Kubernetes Security#

Network policies enforced via @psyche/k8s-security library
GPU nodes isolated from public internet
Pod security contexts restrict privilege escalation

12. Observability Architecture#

Psyche instruments all services with a consistent three-layer observability stack: distributed traces for request-level visibility, Prometheus metrics for aggregate health, and structured logs correlated by trace IDs. The @psyche/slo-alerting library enforces per-component latency budgets and fires alerts before the full 500ms budget is exhausted.

Layer	Tool	Data Captured
Tracing	OpenTelemetry → Jaeger	Distributed traces across all service boundaries
Metrics	Prometheus + Grafana	Latency per component, session counts, GPU utilization
Logs	structlog → Loki	Structured JSON with trace correlation
SLO alerts	`@psyche/slo-alerting`	Latency budget violations, error rate thresholds

Key SLO Metrics#

The following Prometheus metric names and their targets define the system's SLOs. Any breach fires an alert via @psyche/slo-alerting.

text

psyche_session_e2e_latency_ms          (target: < 500 ms p95)
psyche_avatar_frame_latency_ms         (target: < 33 ms p99)
psyche_tts_first_byte_latency_ms       (target: < 100 ms p95)
psyche_stt_transcription_latency_ms    (target: < 100 ms p95)
psyche_llm_response_latency_ms         (target: < 200 ms p95)
psyche_session_error_rate              (target: < 1%)
psyche_gpu_utilization_percent         (alerting threshold: > 85%)

13. Deployment Architecture#

Kubernetes Topology#

The deployment separates CPU-bound services (Deployments, horizontally scalable) from GPU-bound engines (DaemonSets on dedicated GPU node pools). Stateful data services run as StatefulSets with persistent volumes, and the nightly memory consolidation job runs as a Kubernetes CronJob.

text

Ingress
  └── psyche-api-gateway         (Deployment, 2–10 replicas)
       ├── psyche-orchestrator    (Deployment, 2–8 replicas)
       ├── psyche-persona-service (Deployment, 2–4 replicas)
       ├── psyche-knowledge       (Deployment, 2–4 replicas)
       ├── psyche-conferencing    (Deployment, 2–8 replicas)
       └── psyche-computer-use   (Deployment, 2–4 replicas)

GPU Node Pool:
  ├── psyche-avatar-engine       (DaemonSet on GPU nodes, CUDA)
  ├── psyche-voice-engine        (DaemonSet on GPU nodes, CUDA)
  └── psyche-perception-engine   (DaemonSet on GPU nodes, CUDA)

StatefulSets:
  ├── postgres-psyche
  ├── redis-psyche
  └── qdrant-psyche

Jobs:
  └── psyche-memory-consolidator (CronJob, nightly)

Admin Dashboard#

Next.js app deployed to CDN
API calls proxied through API Gateway
Available at http://localhost:3000 in development

14. Domain Dependencies#

External Services Psyche Depends On#

These third-party services are essential to Psyche's real-time capabilities. Each is pluggable — if a provider is unavailable, Psyche falls back to the next option in the chain rather than failing the session.

Service	Purpose
Anthropic Claude	LLM for conversation, computer use vision
ElevenLabs	Primary TTS voice synthesis
Deepgram	Primary real-time STT
Voyage AI	Knowledge base embeddings
Qdrant	Vector similarity search
Zoom / Teams / Meet / Webex	Video conferencing platforms
Tavus	Optional cloud avatar rendering (Phoenix)

Oshun Domain Dependencies#

Psyche draws on two other Oshun domains to augment its knowledge capabilities. The boundary is clear: Psyche owns the real-time assistant behavior and rendering; Sophia owns the research and academic knowledge infrastructure that Psyche taps for knowledge-intensive persona configurations. Iris provides the conversational AI capabilities that power general assistant use cases within Psyche sessions.

Domain	Library	Purpose
Iris	`@iris/integrations-psyche`	Conversation AI capabilities
Sophia	`@psyche/sophia-integration`	Knowledge enrichment via research search

Domains That Depend on Psyche#

Psyche is a standalone product domain. Other domains do not currently depend on it directly, though Iris's integration library (@iris/integrations-psyche) references Psyche APIs for virtual assistant use cases.