Hyper-Realistic AI Virtual Assistant Platform
Psyche (codename: Serwaa) is the AI-powered digital human platform within the
Oshun monorepo. It creates photorealistic virtual assistants capable of
participating in video conferences indistinguishably from real humans. The
platform combines real-time 3D avatar rendering (Gaussian Splatting), neural
voice synthesis, conversational AI (Claude), computer use capabilities, and
native integration with Zoom, Microsoft Teams, Google Meet, and Webex.
Table of Contents#
Key Concepts
Performance Targets
Applications
Libraries
Technology Stack
Architecture Overview
Quick Start
Related Documentation
Key Concepts#
Personas#
A persona defines the identity, personality, voice, and appearance of a
virtual assistant. Each persona includes:
Personality traits -- formality, friendliness, verbosity, humor, empathy,
assertiveness (each a 0.0-1.0 scale)
Voice configuration -- provider (ElevenLabs, XTTS, Cartesia, OpenAI),
speed, pitch, custom voice cloning
Avatar -- 3D Gaussian Splatting model, style, background, camera position
Expertise tags -- domain knowledge areas
Tools -- registered API/function/webhook/MCP tools the persona can invoke
Sessions#
A session represents a single interaction between a virtual assistant and
human participants. Sessions manage the lifecycle of joining a video conference,
rendering the avatar, synthesizing speech, processing perception, and executing
tools. Session states: pending -> connecting -> active -> paused -> ended.
Knowledge Base#
The knowledge base provides RAG (Retrieval-Augmented Generation)
capabilities. Documents are uploaded, chunked, embedded into vectors (stored in
Qdrant), and retrieved via semantic search during conversations to ground the
AI's responses in factual, domain-specific information.
Computer Use#
Psyche integrates Claude's computer use capabilities, allowing the virtual
assistant to operate a sandboxed browser, analyze screens, and execute tasks on
behalf of the user during a video conference.
Metric
Target
Avatar rendering
<33ms (30+ FPS)
Voice synthesis
<100ms latency
End-to-end response
<500ms
Video quality
1080p @ 30 FPS
Audio quality
48kHz, 16-bit
Applications#
Admin Dashboard (apps/psyche/admin)#
A Next.js web application for managing the Psyche platform. Provides CRUD
operations for personas, sessions, knowledge base documents, tools, and system
configuration.
Attribute
Value
Framework
Next.js
Port
3000
Tags
scope:psyche, type:nextjs, platform:web
Build
pnpm nx build psyche-admin
Dev
pnpm nx dev psyche-admin
Test
pnpm nx test psyche-admin
Python Backend Services#
The core platform runs as Python FastAPI microservices. These are located in the
serwaa/ directory (pre-migration location) and are being migrated into the
libs/psyche/ structure as TypeScript libraries where appropriate.
Service
Port
GPU Required
Description
API Gateway
8000
No
REST/WebSocket entry point, auth, routing
Avatar Engine
8001
Yes (8GB)
3D Gaussian Splatting, FLAME model, lip sync
Voice Engine
8002
Yes (6GB)
TTS (XTTS v2), STT (Whisper), voice cloning
Behavior Engine
8003
No
FACS expressions, gaze control, head movement
Perception Engine
8004
Yes (4GB)
Face detection, emotion recognition, gaze
Computer Use Engine
8005
No
Browser automation, screen analysis, sandbox
Conferencing Gateway
8006
No
Zoom/Teams/Meet/Webex SDK integration
Session Orchestrator
8007
No
Session state, pipeline coordination
Knowledge Base
8008
No
Document ingestion, vector embeddings, search
Persona Service
8009
No
Identity management, personality config
Learning System
8010
No
Continuous learning and adaptation
Libraries#
The Psyche domain contains 100+ TypeScript libraries under libs/psyche/. They
are organized by functional area.
Core Infrastructure#
Library
Description
common
Shared types, utilities, and constants
auth
Authentication and authorization
database
Database schema and migrations
cache
Redis-based caching layer
storage
S3-compatible file storage
logging
Structured logging
tracing
Distributed tracing
messaging
Inter-service messaging
Avatar System#
Library
Description
avatar-core
Core avatar rendering pipeline
avatar-cache
Avatar model caching and preloading
avatar-expressions
Facial expression system
avatar-lipsync
Lip synchronization with speech
avatar-quality
Quality metrics and optimization
avatar-taa
Temporal anti-aliasing for avatar rendering
avatar-training
Avatar model training pipeline
avatar-isis-integration
Cross-domain avatar generation via Isis
Voice and Speech#
Library
Description
voice-synthesis
Text-to-speech synthesis
voice-streaming
Real-time voice streaming
voice-consistency
Voice quality consistency monitoring
speech-recognition
Speech-to-text recognition
viseme-generator
Visual phoneme generation for lip sync
noise-handling
Noise reduction and audio preprocessing
Behavior and Expression#
Library
Description
emotion-engine
Emotional response generation
emotion-recognition
User emotion detection from video/audio
micro-expressions
Subtle facial micro-expression system
facs-expressions
FACS (Facial Action Coding System) control
gaze-control
Eye gaze direction management
gaze-estimation
User gaze tracking estimation
head-movement
Natural head motion generation
head-pose-estimation
User head pose detection
posture-system
Body posture management
gesture-system
Hand and body gesture system
behavior-coordinator
Multi-channel behavior orchestration
behavior-anomaly-detector
Behavioral anomaly detection
uncanny-valley
Uncanny valley effect mitigation
Perception#
Library
Description
face-detection
Face detection in video streams
face-analysis
Facial feature analysis
eye-tracking
Eye tracking from camera feed
engagement-detector
User engagement level measurement
attention-tracker
Participant attention tracking
participation-tracker
Meeting participation analysis
Memory System#
Library
Description
memory-core
Core memory management
memory-working
Short-term working memory
memory-in-context
In-context memory for conversations
memory-embeddings
Memory vector embedding generation
memory-retrieval
Memory retrieval and recall
memory-archival
Long-term memory archival
memory-consolidation
Memory consolidation and summarization
memory-persistence
Memory persistence to database
memory-tools
Memory-related tool implementations
Knowledge and Intelligence#
Library
Description
knowledge-context
Knowledge context management
knowledge-ingestion
Document ingestion pipeline
knowledge-retrieval
RAG retrieval for conversations
dialogue-manager
Conversation flow management
Computer Use#
Library
Description
computer-use-core
Core computer use orchestration
browser-automation
Sandboxed browser control
screen-analysis
Screen content understanding
sandbox
Secure execution sandbox
tool-registry
Tool registration and management
tool-execution
Tool execution engine
action-safety
Action safety validation
Video Conferencing Integration#
Library
Description
conferencing-core
Core conferencing abstraction
zoom-integration
Zoom SDK integration
teams-integration
Microsoft Teams SDK integration
meet-integration
Google Meet WebRTC integration
webex-integration
Webex SDK integration
participant-manager
Meeting participant management
state-sync
Conferencing state synchronization
Tavus Integration#
Library
Description
tavus-bridge
Tavus platform bridge
tavus-client
Tavus API client
tavus-conversation
Tavus conversational interface
tavus-expression
Tavus expression control
tavus-hybrid
Hybrid Tavus/native rendering
tavus-knowledge
Tavus knowledge integration
tavus-llm
Tavus LLM orchestration
tavus-memories
Tavus memory system integration
tavus-objectives
Tavus objective management
tavus-perception
Tavus perception pipeline
tavus-persona-manager
Tavus persona management
tavus-phoenix
Tavus Phoenix integration
tavus-pipecat
Tavus Pipecat pipeline integration
tavus-pipeline
Tavus rendering pipeline
tavus-replica-manager
Tavus replica management
tavus-tools
Tavus tool integration
tavus-tts
Tavus text-to-speech integration
tavus-turntaking
Tavus turn-taking management
Observability and Quality#
Library
Description
latency-analysis
End-to-end latency measurement
slo-alerting
SLO monitoring and alerting
nps
Net Promoter Score tracking
satisfaction
User satisfaction measurement
human-evaluation
Human evaluation framework
k8s-security
Kubernetes security policies
accessibility
Accessibility features
Cross-Domain Integration#
Library
Description
sophia-integration
Sophia knowledge/search integration
sophia-search-integration
Sophia semantic search bridge
recall-integration
Recall memory system bridge
Technology Stack#
Component
Technology
Backend Services
Python 3.11+, FastAPI, uvicorn
Admin Dashboard
TypeScript, Next.js, Tailwind CSS
TypeScript Libs
TypeScript, Node.js, Vitest
Python Tests
pytest, pytest-asyncio, hypothesis
Avatar Rendering
3D Gaussian Splatting, FLAME model, CUDA
Voice Synthesis
XTTS v2, ElevenLabs, Cartesia, OpenAI TTS
Speech Recognition
Whisper
LLM
Claude (Anthropic)
Database
PostgreSQL (pgvector)
Vector Store
Qdrant
Cache
Redis
Object Storage
MinIO (S3-compatible)
Build System
Nx, pnpm (TS), Poetry (Python)
Code Quality
ESLint, Prettier (TS), Ruff, mypy (Python)
Conferencing
Zoom SDK, Teams SDK, WebRTC, Webex SDK
Package Manager
pnpm 10+ (TS), Poetry 1.7+ (Python)
Architecture Overview#
text Copy
┌───────────────────────────────────────────────────────────────────┐
│ EXTERNAL CLIENTS │
│ Web App | Mobile App | REST API | Zoom/Teams/Meet/Webex │
└──────────────────────────────┬────────────────────────────────────┘
│
▼
┌──────────────────────┐
│ API Gateway :8000 │
│ (Auth, Rate Limit) │
└──────────┬───────────┘
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Orchestrator │ │ Persona Service │ │ Knowledge Base │
│ :8007 │ │ :8009 │ │ :8008 │
└────────┬─────────┘ └──────────────────┘ └──────────────────┘
│
│ Session Coordination
▼
┌─────────────────────────────────────────────────────────────┐
│ REAL-TIME PROCESSING LAYER │
│ │
│ Avatar Engine :8001 │ Voice Engine :8002 │ Behavior :8003 │
│ (GPU: 8GB VRAM) │ (GPU: 6GB VRAM) │ (CPU only) │
│ 3D Gaussian Splat. │ TTS/STT/Cloning │ FACS/Gaze/Head │
│ │
│ Perception :8004 │ Conferencing :8006 │ Computer Use │
│ (GPU: 4GB VRAM) │ Zoom/Teams/Meet │ :8005 │
│ Face/Emotion/Gaze │ Webex integration │ Browser/Screen │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ DATA LAYER │
│ PostgreSQL (pgvector) │ Redis │ Qdrant │ MinIO (S3) │
│ Personas, Sessions │ Cache │ Vector │ ML Models, Avatars │
└─────────────────────────────────────────────────────────────┘
For detailed architecture diagrams, service interactions, and data flows, see
architecture.md .
Quick Start#
Prerequisites#
Node.js >= 20.0.0, pnpm >= 10.0
Python 3.11+, Poetry 1.7+
Docker >= 24.0
NVIDIA GPU with CUDA 12.1+ (for avatar/voice engines)
Setup#
bash Copy
# Clone and install
git clone git@github.com:GreyChimp/oshun.git
cd oshun
pnpm install
# Start infrastructure
docker compose -f docker/docker-compose.dev.yml up -d
# Start Qdrant for vector search
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d
# Start the Admin Dashboard
pnpm nx dev psyche-admin
# Available at http://localhost:3000
# Start Python backend services (from serwaa/ directory)
cd serwaa && poetry install
poetry run uvicorn services.api_gateway.main:app --reload --port 8000
For the complete development setup guide including GPU configuration, Python
environment setup, and running all services, see
developer-guide.md .