Eve library

Psyche Domain

Guide

Revision 1

On this page

Hyper-Realistic AI Virtual Assistant Platform

Psyche (codename: Serwaa) is the AI-powered digital human platform within the Oshun monorepo. It creates photorealistic virtual assistants capable of participating in video conferences indistinguishably from real humans. The platform combines real-time 3D avatar rendering (Gaussian Splatting), neural voice synthesis, conversational AI (Claude), computer use capabilities, and native integration with Zoom, Microsoft Teams, Google Meet, and Webex.


Table of Contents#

  1. Key Concepts
  2. Performance Targets
  3. Applications
  4. Libraries
  5. Technology Stack
  6. Architecture Overview
  7. Quick Start
  8. Related Documentation

Key Concepts#

Personas#

A persona defines the identity, personality, voice, and appearance of a virtual assistant. Each persona includes:

  • Personality traits -- formality, friendliness, verbosity, humor, empathy, assertiveness (each a 0.0-1.0 scale)
  • Voice configuration -- provider (ElevenLabs, XTTS, Cartesia, OpenAI), speed, pitch, custom voice cloning
  • Avatar -- 3D Gaussian Splatting model, style, background, camera position
  • Expertise tags -- domain knowledge areas
  • Tools -- registered API/function/webhook/MCP tools the persona can invoke

Sessions#

A session represents a single interaction between a virtual assistant and human participants. Sessions manage the lifecycle of joining a video conference, rendering the avatar, synthesizing speech, processing perception, and executing tools. Session states: pending -> connecting -> active -> paused -> ended.

Knowledge Base#

The knowledge base provides RAG (Retrieval-Augmented Generation) capabilities. Documents are uploaded, chunked, embedded into vectors (stored in Qdrant), and retrieved via semantic search during conversations to ground the AI's responses in factual, domain-specific information.

Computer Use#

Psyche integrates Claude's computer use capabilities, allowing the virtual assistant to operate a sandboxed browser, analyze screens, and execute tasks on behalf of the user during a video conference.


Performance Targets#

Metric Target
Avatar rendering <33ms (30+ FPS)
Voice synthesis <100ms latency
End-to-end response <500ms
Video quality 1080p @ 30 FPS
Audio quality 48kHz, 16-bit

Applications#

Admin Dashboard (apps/psyche/admin)#

A Next.js web application for managing the Psyche platform. Provides CRUD operations for personas, sessions, knowledge base documents, tools, and system configuration.

Attribute Value
Framework Next.js
Port 3000
Tags scope:psyche, type:nextjs, platform:web
Build pnpm nx build psyche-admin
Dev pnpm nx dev psyche-admin
Test pnpm nx test psyche-admin

Python Backend Services#

The core platform runs as Python FastAPI microservices. These are located in the serwaa/ directory (pre-migration location) and are being migrated into the libs/psyche/ structure as TypeScript libraries where appropriate.

Service Port GPU Required Description
API Gateway 8000 No REST/WebSocket entry point, auth, routing
Avatar Engine 8001 Yes (8GB) 3D Gaussian Splatting, FLAME model, lip sync
Voice Engine 8002 Yes (6GB) TTS (XTTS v2), STT (Whisper), voice cloning
Behavior Engine 8003 No FACS expressions, gaze control, head movement
Perception Engine 8004 Yes (4GB) Face detection, emotion recognition, gaze
Computer Use Engine 8005 No Browser automation, screen analysis, sandbox
Conferencing Gateway 8006 No Zoom/Teams/Meet/Webex SDK integration
Session Orchestrator 8007 No Session state, pipeline coordination
Knowledge Base 8008 No Document ingestion, vector embeddings, search
Persona Service 8009 No Identity management, personality config
Learning System 8010 No Continuous learning and adaptation

Libraries#

The Psyche domain contains 100+ TypeScript libraries under libs/psyche/. They are organized by functional area.

Core Infrastructure#

Library Description
common Shared types, utilities, and constants
auth Authentication and authorization
database Database schema and migrations
cache Redis-based caching layer
storage S3-compatible file storage
logging Structured logging
tracing Distributed tracing
messaging Inter-service messaging

Avatar System#

Library Description
avatar-core Core avatar rendering pipeline
avatar-cache Avatar model caching and preloading
avatar-expressions Facial expression system
avatar-lipsync Lip synchronization with speech
avatar-quality Quality metrics and optimization
avatar-taa Temporal anti-aliasing for avatar rendering
avatar-training Avatar model training pipeline
avatar-isis-integration Cross-domain avatar generation via Isis

Voice and Speech#

Library Description
voice-synthesis Text-to-speech synthesis
voice-streaming Real-time voice streaming
voice-consistency Voice quality consistency monitoring
speech-recognition Speech-to-text recognition
viseme-generator Visual phoneme generation for lip sync
noise-handling Noise reduction and audio preprocessing

Behavior and Expression#

Library Description
emotion-engine Emotional response generation
emotion-recognition User emotion detection from video/audio
micro-expressions Subtle facial micro-expression system
facs-expressions FACS (Facial Action Coding System) control
gaze-control Eye gaze direction management
gaze-estimation User gaze tracking estimation
head-movement Natural head motion generation
head-pose-estimation User head pose detection
posture-system Body posture management
gesture-system Hand and body gesture system
behavior-coordinator Multi-channel behavior orchestration
behavior-anomaly-detector Behavioral anomaly detection
uncanny-valley Uncanny valley effect mitigation

Perception#

Library Description
face-detection Face detection in video streams
face-analysis Facial feature analysis
eye-tracking Eye tracking from camera feed
engagement-detector User engagement level measurement
attention-tracker Participant attention tracking
participation-tracker Meeting participation analysis

Memory System#

Library Description
memory-core Core memory management
memory-working Short-term working memory
memory-in-context In-context memory for conversations
memory-embeddings Memory vector embedding generation
memory-retrieval Memory retrieval and recall
memory-archival Long-term memory archival
memory-consolidation Memory consolidation and summarization
memory-persistence Memory persistence to database
memory-tools Memory-related tool implementations

Knowledge and Intelligence#

Library Description
knowledge-context Knowledge context management
knowledge-ingestion Document ingestion pipeline
knowledge-retrieval RAG retrieval for conversations
dialogue-manager Conversation flow management

Computer Use#

Library Description
computer-use-core Core computer use orchestration
browser-automation Sandboxed browser control
screen-analysis Screen content understanding
sandbox Secure execution sandbox
tool-registry Tool registration and management
tool-execution Tool execution engine
action-safety Action safety validation

Video Conferencing Integration#

Library Description
conferencing-core Core conferencing abstraction
zoom-integration Zoom SDK integration
teams-integration Microsoft Teams SDK integration
meet-integration Google Meet WebRTC integration
webex-integration Webex SDK integration
participant-manager Meeting participant management
state-sync Conferencing state synchronization

Tavus Integration#

Library Description
tavus-bridge Tavus platform bridge
tavus-client Tavus API client
tavus-conversation Tavus conversational interface
tavus-expression Tavus expression control
tavus-hybrid Hybrid Tavus/native rendering
tavus-knowledge Tavus knowledge integration
tavus-llm Tavus LLM orchestration
tavus-memories Tavus memory system integration
tavus-objectives Tavus objective management
tavus-perception Tavus perception pipeline
tavus-persona-manager Tavus persona management
tavus-phoenix Tavus Phoenix integration
tavus-pipecat Tavus Pipecat pipeline integration
tavus-pipeline Tavus rendering pipeline
tavus-replica-manager Tavus replica management
tavus-tools Tavus tool integration
tavus-tts Tavus text-to-speech integration
tavus-turntaking Tavus turn-taking management

Observability and Quality#

Library Description
latency-analysis End-to-end latency measurement
slo-alerting SLO monitoring and alerting
nps Net Promoter Score tracking
satisfaction User satisfaction measurement
human-evaluation Human evaluation framework
k8s-security Kubernetes security policies
accessibility Accessibility features

Cross-Domain Integration#

Library Description
sophia-integration Sophia knowledge/search integration
sophia-search-integration Sophia semantic search bridge
recall-integration Recall memory system bridge

Technology Stack#

Component Technology
Backend Services Python 3.11+, FastAPI, uvicorn
Admin Dashboard TypeScript, Next.js, Tailwind CSS
TypeScript Libs TypeScript, Node.js, Vitest
Python Tests pytest, pytest-asyncio, hypothesis
Avatar Rendering 3D Gaussian Splatting, FLAME model, CUDA
Voice Synthesis XTTS v2, ElevenLabs, Cartesia, OpenAI TTS
Speech Recognition Whisper
LLM Claude (Anthropic)
Database PostgreSQL (pgvector)
Vector Store Qdrant
Cache Redis
Object Storage MinIO (S3-compatible)
Build System Nx, pnpm (TS), Poetry (Python)
Code Quality ESLint, Prettier (TS), Ruff, mypy (Python)
Conferencing Zoom SDK, Teams SDK, WebRTC, Webex SDK
Package Manager pnpm 10+ (TS), Poetry 1.7+ (Python)

Architecture Overview#

text
┌───────────────────────────────────────────────────────────────────┐
│                        EXTERNAL CLIENTS                            │
│  Web App  |  Mobile App  |  REST API  |  Zoom/Teams/Meet/Webex    │
└──────────────────────────────┬────────────────────────────────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │   API Gateway :8000   │
                    │  (Auth, Rate Limit)   │
                    └──────────┬───────────┘
                               │
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│   Orchestrator   │ │  Persona Service │ │  Knowledge Base  │
│     :8007        │ │     :8009        │ │     :8008        │
└────────┬─────────┘ └──────────────────┘ └──────────────────┘
         │
         │  Session Coordination
         ▼
┌─────────────────────────────────────────────────────────────┐
│                  REAL-TIME PROCESSING LAYER                    │
│                                                                │
│  Avatar Engine :8001  │  Voice Engine :8002  │  Behavior :8003 │
│  (GPU: 8GB VRAM)      │  (GPU: 6GB VRAM)    │  (CPU only)     │
│  3D Gaussian Splat.   │  TTS/STT/Cloning    │  FACS/Gaze/Head │
│                                                                │
│  Perception :8004     │  Conferencing :8006  │  Computer Use   │
│  (GPU: 4GB VRAM)      │  Zoom/Teams/Meet     │     :8005       │
│  Face/Emotion/Gaze    │  Webex integration   │  Browser/Screen │
└─────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                        DATA LAYER                              │
│  PostgreSQL (pgvector) │ Redis │ Qdrant │ MinIO (S3)          │
│  Personas, Sessions    │ Cache │ Vector │ ML Models, Avatars  │
└─────────────────────────────────────────────────────────────┘

For detailed architecture diagrams, service interactions, and data flows, see architecture.md.


Quick Start#

Prerequisites#

  • Node.js >= 20.0.0, pnpm >= 10.0
  • Python 3.11+, Poetry 1.7+
  • Docker >= 24.0
  • NVIDIA GPU with CUDA 12.1+ (for avatar/voice engines)

Setup#

bash
# Clone and install
git clone git@github.com:GreyChimp/oshun.git
cd oshun
pnpm install

# Start infrastructure
docker compose -f docker/docker-compose.dev.yml up -d
# Start Qdrant for vector search
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Start the Admin Dashboard
pnpm nx dev psyche-admin
# Available at http://localhost:3000

# Start Python backend services (from serwaa/ directory)
cd serwaa && poetry install
poetry run uvicorn services.api_gateway.main:app --reload --port 8000

For the complete development setup guide including GPU configuration, Python environment setup, and running all services, see developer-guide.md.