# Psyche Domain

> Hyper-Realistic AI Virtual Assistant Platform

Psyche (codename: Serwaa) is the AI-powered digital human platform within the
Oshun monorepo. It creates photorealistic virtual assistants capable of
participating in video conferences indistinguishably from real humans. The
platform combines real-time 3D avatar rendering (Gaussian Splatting), neural
voice synthesis, conversational AI (Claude), computer use capabilities, and
native integration with Zoom, Microsoft Teams, Google Meet, and Webex.

---

## Table of Contents

1. [Key Concepts](#key-concepts)
2. [Performance Targets](#performance-targets)
3. [Applications](#applications)
4. [Libraries](#libraries)
5. [Technology Stack](#technology-stack)
6. [Architecture Overview](#architecture-overview)
7. [Quick Start](#quick-start)
8. [Related Documentation](#related-documentation)

---

## Key Concepts

### Personas

A **persona** defines the identity, personality, voice, and appearance of a
virtual assistant. Each persona includes:

- **Personality traits** -- formality, friendliness, verbosity, humor, empathy,
  assertiveness (each a 0.0-1.0 scale)
- **Voice configuration** -- provider (ElevenLabs, XTTS, Cartesia, OpenAI),
  speed, pitch, custom voice cloning
- **Avatar** -- 3D Gaussian Splatting model, style, background, camera position
- **Expertise tags** -- domain knowledge areas
- **Tools** -- registered API/function/webhook/MCP tools the persona can invoke

### Sessions

A **session** represents a single interaction between a virtual assistant and
human participants. Sessions manage the lifecycle of joining a video conference,
rendering the avatar, synthesizing speech, processing perception, and executing
tools. Session states: `pending -> connecting -> active -> paused -> ended`.

### Knowledge Base

The **knowledge base** provides RAG (Retrieval-Augmented Generation)
capabilities. Documents are uploaded, chunked, embedded into vectors (stored in
Qdrant), and retrieved via semantic search during conversations to ground the
AI's responses in factual, domain-specific information.

### Computer Use

Psyche integrates Claude's **computer use** capabilities, allowing the virtual
assistant to operate a sandboxed browser, analyze screens, and execute tasks on
behalf of the user during a video conference.

---

## Performance Targets

| Metric              | Target          |
| ------------------- | --------------- |
| Avatar rendering    | <33ms (30+ FPS) |
| Voice synthesis     | <100ms latency  |
| End-to-end response | <500ms          |
| Video quality       | 1080p @ 30 FPS  |
| Audio quality       | 48kHz, 16-bit   |

---

## Applications

### Admin Dashboard (`apps/psyche/admin`)

A Next.js web application for managing the Psyche platform. Provides CRUD
operations for personas, sessions, knowledge base documents, tools, and system
configuration.

| Attribute | Value                                         |
| --------- | --------------------------------------------- |
| Framework | Next.js                                       |
| Port      | 3000                                          |
| Tags      | `scope:psyche`, `type:nextjs`, `platform:web` |
| Build     | `pnpm nx build psyche-admin`                  |
| Dev       | `pnpm nx dev psyche-admin`                    |
| Test      | `pnpm nx test psyche-admin`                   |

### Python Backend Services

The core platform runs as Python FastAPI microservices. These are located in the
`serwaa/` directory (pre-migration location) and are being migrated into the
`libs/psyche/` structure as TypeScript libraries where appropriate.

| Service              | Port | GPU Required | Description                                   |
| -------------------- | ---- | ------------ | --------------------------------------------- |
| API Gateway          | 8000 | No           | REST/WebSocket entry point, auth, routing     |
| Avatar Engine        | 8001 | Yes (8GB)    | 3D Gaussian Splatting, FLAME model, lip sync  |
| Voice Engine         | 8002 | Yes (6GB)    | TTS (XTTS v2), STT (Whisper), voice cloning   |
| Behavior Engine      | 8003 | No           | FACS expressions, gaze control, head movement |
| Perception Engine    | 8004 | Yes (4GB)    | Face detection, emotion recognition, gaze     |
| Computer Use Engine  | 8005 | No           | Browser automation, screen analysis, sandbox  |
| Conferencing Gateway | 8006 | No           | Zoom/Teams/Meet/Webex SDK integration         |
| Session Orchestrator | 8007 | No           | Session state, pipeline coordination          |
| Knowledge Base       | 8008 | No           | Document ingestion, vector embeddings, search |
| Persona Service      | 8009 | No           | Identity management, personality config       |
| Learning System      | 8010 | No           | Continuous learning and adaptation            |

---

## Libraries

The Psyche domain contains 100+ TypeScript libraries under `libs/psyche/`. They
are organized by functional area.

### Core Infrastructure

| Library     | Description                            |
| ----------- | -------------------------------------- |
| `common`    | Shared types, utilities, and constants |
| `auth`      | Authentication and authorization       |
| `database`  | Database schema and migrations         |
| `cache`     | Redis-based caching layer              |
| `storage`   | S3-compatible file storage             |
| `logging`   | Structured logging                     |
| `tracing`   | Distributed tracing                    |
| `messaging` | Inter-service messaging                |

### Avatar System

| Library                   | Description                                 |
| ------------------------- | ------------------------------------------- |
| `avatar-core`             | Core avatar rendering pipeline              |
| `avatar-cache`            | Avatar model caching and preloading         |
| `avatar-expressions`      | Facial expression system                    |
| `avatar-lipsync`          | Lip synchronization with speech             |
| `avatar-quality`          | Quality metrics and optimization            |
| `avatar-taa`              | Temporal anti-aliasing for avatar rendering |
| `avatar-training`         | Avatar model training pipeline              |
| `avatar-isis-integration` | Cross-domain avatar generation via Isis     |

### Voice and Speech

| Library              | Description                             |
| -------------------- | --------------------------------------- |
| `voice-synthesis`    | Text-to-speech synthesis                |
| `voice-streaming`    | Real-time voice streaming               |
| `voice-consistency`  | Voice quality consistency monitoring    |
| `speech-recognition` | Speech-to-text recognition              |
| `viseme-generator`   | Visual phoneme generation for lip sync  |
| `noise-handling`     | Noise reduction and audio preprocessing |

### Behavior and Expression

| Library                     | Description                                |
| --------------------------- | ------------------------------------------ |
| `emotion-engine`            | Emotional response generation              |
| `emotion-recognition`       | User emotion detection from video/audio    |
| `micro-expressions`         | Subtle facial micro-expression system      |
| `facs-expressions`          | FACS (Facial Action Coding System) control |
| `gaze-control`              | Eye gaze direction management              |
| `gaze-estimation`           | User gaze tracking estimation              |
| `head-movement`             | Natural head motion generation             |
| `head-pose-estimation`      | User head pose detection                   |
| `posture-system`            | Body posture management                    |
| `gesture-system`            | Hand and body gesture system               |
| `behavior-coordinator`      | Multi-channel behavior orchestration       |
| `behavior-anomaly-detector` | Behavioral anomaly detection               |
| `uncanny-valley`            | Uncanny valley effect mitigation           |

### Perception

| Library                 | Description                       |
| ----------------------- | --------------------------------- |
| `face-detection`        | Face detection in video streams   |
| `face-analysis`         | Facial feature analysis           |
| `eye-tracking`          | Eye tracking from camera feed     |
| `engagement-detector`   | User engagement level measurement |
| `attention-tracker`     | Participant attention tracking    |
| `participation-tracker` | Meeting participation analysis    |

### Memory System

| Library                | Description                            |
| ---------------------- | -------------------------------------- |
| `memory-core`          | Core memory management                 |
| `memory-working`       | Short-term working memory              |
| `memory-in-context`    | In-context memory for conversations    |
| `memory-embeddings`    | Memory vector embedding generation     |
| `memory-retrieval`     | Memory retrieval and recall            |
| `memory-archival`      | Long-term memory archival              |
| `memory-consolidation` | Memory consolidation and summarization |
| `memory-persistence`   | Memory persistence to database         |
| `memory-tools`         | Memory-related tool implementations    |

### Knowledge and Intelligence

| Library               | Description                     |
| --------------------- | ------------------------------- |
| `knowledge-context`   | Knowledge context management    |
| `knowledge-ingestion` | Document ingestion pipeline     |
| `knowledge-retrieval` | RAG retrieval for conversations |
| `dialogue-manager`    | Conversation flow management    |

### Computer Use

| Library              | Description                      |
| -------------------- | -------------------------------- |
| `computer-use-core`  | Core computer use orchestration  |
| `browser-automation` | Sandboxed browser control        |
| `screen-analysis`    | Screen content understanding     |
| `sandbox`            | Secure execution sandbox         |
| `tool-registry`      | Tool registration and management |
| `tool-execution`     | Tool execution engine            |
| `action-safety`      | Action safety validation         |

### Video Conferencing Integration

| Library               | Description                        |
| --------------------- | ---------------------------------- |
| `conferencing-core`   | Core conferencing abstraction      |
| `zoom-integration`    | Zoom SDK integration               |
| `teams-integration`   | Microsoft Teams SDK integration    |
| `meet-integration`    | Google Meet WebRTC integration     |
| `webex-integration`   | Webex SDK integration              |
| `participant-manager` | Meeting participant management     |
| `state-sync`          | Conferencing state synchronization |

### Tavus Integration

| Library                 | Description                        |
| ----------------------- | ---------------------------------- |
| `tavus-bridge`          | Tavus platform bridge              |
| `tavus-client`          | Tavus API client                   |
| `tavus-conversation`    | Tavus conversational interface     |
| `tavus-expression`      | Tavus expression control           |
| `tavus-hybrid`          | Hybrid Tavus/native rendering      |
| `tavus-knowledge`       | Tavus knowledge integration        |
| `tavus-llm`             | Tavus LLM orchestration            |
| `tavus-memories`        | Tavus memory system integration    |
| `tavus-objectives`      | Tavus objective management         |
| `tavus-perception`      | Tavus perception pipeline          |
| `tavus-persona-manager` | Tavus persona management           |
| `tavus-phoenix`         | Tavus Phoenix integration          |
| `tavus-pipecat`         | Tavus Pipecat pipeline integration |
| `tavus-pipeline`        | Tavus rendering pipeline           |
| `tavus-replica-manager` | Tavus replica management           |
| `tavus-tools`           | Tavus tool integration             |
| `tavus-tts`             | Tavus text-to-speech integration   |
| `tavus-turntaking`      | Tavus turn-taking management       |

### Observability and Quality

| Library            | Description                    |
| ------------------ | ------------------------------ |
| `latency-analysis` | End-to-end latency measurement |
| `slo-alerting`     | SLO monitoring and alerting    |
| `nps`              | Net Promoter Score tracking    |
| `satisfaction`     | User satisfaction measurement  |
| `human-evaluation` | Human evaluation framework     |
| `k8s-security`     | Kubernetes security policies   |
| `accessibility`    | Accessibility features         |

### Cross-Domain Integration

| Library                     | Description                         |
| --------------------------- | ----------------------------------- |
| `sophia-integration`        | Sophia knowledge/search integration |
| `sophia-search-integration` | Sophia semantic search bridge       |
| `recall-integration`        | Recall memory system bridge         |

---

## Technology Stack

| Component          | Technology                                 |
| ------------------ | ------------------------------------------ |
| Backend Services   | Python 3.11+, FastAPI, uvicorn             |
| Admin Dashboard    | TypeScript, Next.js, Tailwind CSS          |
| TypeScript Libs    | TypeScript, Node.js, Vitest                |
| Python Tests       | pytest, pytest-asyncio, hypothesis         |
| Avatar Rendering   | 3D Gaussian Splatting, FLAME model, CUDA   |
| Voice Synthesis    | XTTS v2, ElevenLabs, Cartesia, OpenAI TTS  |
| Speech Recognition | Whisper                                    |
| LLM                | Claude (Anthropic)                         |
| Database           | PostgreSQL (pgvector)                      |
| Vector Store       | Qdrant                                     |
| Cache              | Redis                                      |
| Object Storage     | MinIO (S3-compatible)                      |
| Build System       | Nx, pnpm (TS), Poetry (Python)             |
| Code Quality       | ESLint, Prettier (TS), Ruff, mypy (Python) |
| Conferencing       | Zoom SDK, Teams SDK, WebRTC, Webex SDK     |
| Package Manager    | pnpm 10+ (TS), Poetry 1.7+ (Python)        |

---

## Architecture Overview

```
┌───────────────────────────────────────────────────────────────────┐
│                        EXTERNAL CLIENTS                            │
│  Web App  |  Mobile App  |  REST API  |  Zoom/Teams/Meet/Webex    │
└──────────────────────────────┬────────────────────────────────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │   API Gateway :8000   │
                    │  (Auth, Rate Limit)   │
                    └──────────┬───────────┘
                               │
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│   Orchestrator   │ │  Persona Service │ │  Knowledge Base  │
│     :8007        │ │     :8009        │ │     :8008        │
└────────┬─────────┘ └──────────────────┘ └──────────────────┘
         │
         │  Session Coordination
         ▼
┌─────────────────────────────────────────────────────────────┐
│                  REAL-TIME PROCESSING LAYER                    │
│                                                                │
│  Avatar Engine :8001  │  Voice Engine :8002  │  Behavior :8003 │
│  (GPU: 8GB VRAM)      │  (GPU: 6GB VRAM)    │  (CPU only)     │
│  3D Gaussian Splat.   │  TTS/STT/Cloning    │  FACS/Gaze/Head │
│                                                                │
│  Perception :8004     │  Conferencing :8006  │  Computer Use   │
│  (GPU: 4GB VRAM)      │  Zoom/Teams/Meet     │     :8005       │
│  Face/Emotion/Gaze    │  Webex integration   │  Browser/Screen │
└─────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                        DATA LAYER                              │
│  PostgreSQL (pgvector) │ Redis │ Qdrant │ MinIO (S3)          │
│  Personas, Sessions    │ Cache │ Vector │ ML Models, Avatars  │
└─────────────────────────────────────────────────────────────┘
```

For detailed architecture diagrams, service interactions, and data flows, see
[architecture.md](./architecture.md).

---

## Quick Start

### Prerequisites

- Node.js >= 20.0.0, pnpm >= 10.0
- Python 3.11+, Poetry 1.7+
- Docker >= 24.0
- NVIDIA GPU with CUDA 12.1+ (for avatar/voice engines)

### Setup

```bash
# Clone and install
git clone git@github.com:GreyChimp/oshun.git
cd oshun
pnpm install

# Start infrastructure
docker compose -f docker/docker-compose.dev.yml up -d
# Start Qdrant for vector search
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

# Start the Admin Dashboard
pnpm nx dev psyche-admin
# Available at http://localhost:3000

# Start Python backend services (from serwaa/ directory)
cd serwaa && poetry install
poetry run uvicorn services.api_gateway.main:app --reload --port 8000
```

For the complete development setup guide including GPU configuration, Python
environment setup, and running all services, see
[developer-guide.md](./developer-guide.md).

---

## Related Documentation

- [Architecture](./architecture.md) -- System architecture, service topology,
  data flows
- [Developer Guide](./developer-guide.md) -- Local setup, running services,
  testing
- [API Documentation](./api-documentation.md) -- REST, WebSocket, and gRPC API
  reference
- [Data Models](./data-models.md) -- Database schema and data model
  documentation
- [ML Models](./ml-models.md) -- Machine learning model inventory and
  dependencies
- [Integration Guide](./integration-guide.md) -- Integrating with the Psyche
  platform
- [Persona Configuration Guide](./persona-configuration-guide.md) -- Creating
  and configuring personas
- [Admin Dashboard Guide](./admin-dashboard-guide.md) -- Using the admin
  interface
- [Service Inventory](./service-inventory.md) -- Detailed breakdown of all
  services
- [Security Audit](./security-audit.md) -- Security assessment and
  recommendations
- [External Integrations](./external-integrations.md) -- Third-party service
  integrations
- [Zoom Integration Certification](./zoom-integration-certification.md) -- Zoom
  SDK compliance
- [Teams Integration Certification](./teams-integration-certification.md) --
  Teams SDK compliance
- [Runbooks](./runbooks.md) -- Operational procedures and incident response
