Eve library

Psyche — Systems Deep Dive

Guide

Revision 1

On this page

The services/psyche/ area: a Poetry-workspace of Python/FastAPI services that make up the Psyche AI Virtual Assistant Platform — avatar rendering, voice, perception, computer use, conferencing, memory/learning, and the API edge that fronts them.

What this area is#

Psyche is the platform's embodied AI-assistant domain: the stack that turns a persona into a real-time, audio-visual agent that can see, speak, render a photoreal face, act on a desktop, and join a video call. Unlike most of the Oshun monorepo (TypeScript/Nx), this area is Python 3.11+ on FastAPI, managed as a single Poetry workspace. The root services/psyche/pyproject.toml declares the shared dependency constraints and optional extras (ml, voice, vision, avatar, aws, all) so each service installs only the heavy ML/GPU stack it actually needs, and services/psyche/README.md records that the code was migrated from the earlier "Serwaa" codebase.

Every node is a scope:psyche, layer:service, type:python Nx project. Most are FastAPI applications with a uniform target set in their project.json (serve via poetry run uvicorn, build/install via Poetry, lint with ruff+mypy, test with pytest) and a Dockerfile; a few are tagged runtime:fastapi, others platform:backend cross-cutting libraries. GPU posture is encoded in tags too — gpu:required for the avatar engine, gpu:optional for voice/perception/behavior.

The projects compose into a pipeline rather than a flat list. The psyche-api-gateway is the HTTP/WebSocket edge; psyche-orchestrator owns session lifecycle and resource allocation; the four "engines" (avatar/voice/perception/behavior) are the real-time media plane; psyche-persona-service, psyche-knowledge-base, and psyche-learning-system provide identity, retrieval, and memory; psyche-tool-framework and psyche-computer-use give the assistant the ability to call tools and operate a desktop; psyche-conferencing/psyche-video-conferencing inject the avatar into Zoom/Teams/Meet/Webex; and psyche-tavus-integration is an external SOTA-avatar provider path. psyche-security and psyche-error-handling are the cross-cutting resilience/safety libraries the rest depend on.

A note on honesty up front: the area is large and mostly substantively implemented (avatar-engine alone is ~75K LOC of real PyTorch/3DGS code), but it is not uniform. psyche-behavior-engine is an empty scaffold (its logic lives inside the avatar engine), and psyche-video-conferencing is a library-packaged duplicate of the psyche-conferencing source tree. Both are flagged as such in their entries below rather than presented as independent implementations. Many engines also depend on real external providers (Anthropic Computer Use, Tavus, ElevenLabs/Deepgram/Cartesia, Voyage/OpenAI/Cohere) and CUDA GPUs, so "implemented" here means the orchestration and provider-adapter code is real, not that every path runs without creds or hardware.

How it fits the wider system#

Consumers reach Psyche through psyche-api-gateway (REST + WebSocket on port 8000), which is the only public surface; everything else is an internal service the gateway and orchestrator coordinate. Within the area the dependency arrows point toward the engines and the shared libraries: the orchestrator drives the media engines, the engines lean on psyche-security for auth/data-protection and psyche-error-handling for degradation/failover, and the persona/knowledge/ learning services supply the assistant's identity, facts, and long-term memory.

The boundary types Psyche exchanges with the rest of Oshun are defined elsewhere — @psyche/contracts (libs/contracts/psyche) owns the validated wire schemas, covered in the Contracts systems page — so this area focuses on behaviour and runtime rather than on the contract surface. Outward, the heavy integrations are with third-party platforms (conferencing SDKs, avatar/voice/LLM vendors) rather than with sibling Oshun domains, which is why so many project.json files carry platform/provider install-extras and GPU tags.

Entity reference#

psyche-api-gateway#

The REST + WebSocket edge for the whole platform (services/psyche/api-gateway), a real FastAPI app built through the create_app factory in src/api_gateway/app.py (registered middleware, exception handlers, CORS/GZip, ORJSON). It carries a full router set under src/api_gateway/routers/ — sessions, personas, knowledge, tools, webhooks, websocket, experiments, features, and health (liveness/readiness/dependency probes) — backed by SQLAlchemy models in models/, a real-time layer in websocket/ (manager.py, handlers.py), a webhook delivery service (services/webhooks.py, ~630 LOC), and DB migration tooling (db/migrate.py). It is the unified front door the README documents at :8000/docs, and the most integration-heavy node in the area.

psyche-avatar-engine#

The neural-rendering engine and the largest node here (~75K LOC across ~150 files, gpu:required). src/avatar_engine/engine.py defines AvatarEngine, the orchestration class that imports torch, the GaussianModel/FLAMEModel, and a real rasterization/rendering stack (rendering/rasterizer.py, temporal_aa.py, ambient_occlusion.py, dynamic_resolution.py) for 3D Gaussian Splatting avatars. Beyond rendering it spans nerf/, training/, quantization/, lipsync/, expression/, eye/, body/, capture/, culture/, validation/, video/, and a substantial accessibility/ module (description_engine.py ~1.2K LOC, scene_analyzer.py ~0.8K LOC). It also hosts the behavior/ submodule that the behavior-engine service nominally fronts. Real implementation, but predicated on a CUDA GPU and ML extras.

psyche-behavior-engine#

An empty scaffold. The directory contains only README.md, Dockerfile, project.json, pyproject.toml, and .gitkeep placeholders under src/ and tests/ — there is no Python implementation. The README is explicit that "the core behavioral logic is integrated into the Avatar Engine (services/psyche/avatar-engine/src/avatar_engine/behavior/)," so this node is a reserved FACS/expression API slot whose logic actually lives in psyche-avatar-engine. It is documented as a placeholder, not a working service.

psyche-computer-use#

The autonomous desktop-agent service (services/psyche/computer-use, ~23K LOC plus a very large test suite of ~60 files). It implements an observe-reason-act loop around Anthropic's Claude Computer Use API: src/computer_use/agent/ holds loop.py, planning.py, dead_end.py, recovery.py, undo.py, and explain.py, while environment/ provides the Xvfb-sandboxed desktop, vision/ does OCR/UI-element detection, actions/ covers mouse/keyboard/file operations, screen_sharing/ streams via WebRTC, and benchmark/ carries an OSWorld-style evaluation harness (with baselines under benchmark_baselines/). Real agent code, dependent on the Anthropic API and a Playwright/Xvfb runtime.

psyche-conferencing#

The live-meeting injection service (services/psyche/conferencing, FastAPI application served on port 8006; its project.json docker-run target maps /dev/video0 and it exposes install-zoom/install-teams/install-meet Nx install targets). Its video_conferencing package (~113 files, ~96K LOC) provides a platform-agnostic gateway/ adapter plus per-platform integrations zoom/, teams/, meet/, webex/, webrtc/, real virtual_devices/ (v4l2loopback camera + PulseAudio injection), and multiparticipant/, meeting_intelligence/, optimization/, perception/, and accessibility/ modules. This is the runnable conferencing application; psyche-video-conferencing packages the same tree as a library (see below).

psyche-error-handling#

A cross-cutting resilience library (services/psyche/error-handling, platform:backend, ~4K LOC, no FastAPI runtime). It is real and self-contained: src/error_handling/errors/ defines a typed exception hierarchy (types.py, ~584 LOC) and automatic classification (detection.py, ~802 LOC); degradation/__init__.py (~663 LOC) implements multi-level graceful degradation with component-specific degraders; recovery/__init__.py (~858 LOC) carries retry policies, the circuit-breaker pattern, and multi-provider failover; and communication/__init__.py (~678 LOC) handles user-facing error messaging. A genuine implementation other Psyche services import for fault tolerance.

psyche-knowledge-base#

The RAG and document-processing service (services/psyche/knowledge-base, feature:rag, ~14K LOC). src/knowledge_base/ is organised by pipeline stage: ingestion/ (parsers, chunking strategies, cleaning), embedding/, retrieval/, management/, and rag/ (pipeline.py, generator.py, evaluator.py, prompts.py) implementing the retrieve-rank-generate flow with quality evaluation. The README documents multiple embedding providers (Voyage/OpenAI/Cohere/BGE) and vector stores (Pinecone/Qdrant/pgvector/in-memory) behind these adapters; the orchestration is real, individual providers require their respective credentials.

psyche-learning-system#

The memory and skill-acquisition service (services/psyche/learning-system, ~72K LOC, large test suite, main.py FastAPI app). It implements a MemGPT/Letta- style hierarchical memory: src/learning_system/persistent_memory/ holds memory_tools.py, decay_consolidation.py, context_management.py, and persistence.py, complemented by skill_library/ (Voyager-style skill discovery/ composition), knowledge_graph/ (temporal entity-relationship memory), procedural_memory/, instruction_memory/, verbal_instruction/, learning_from_demonstration/, experience_based_learning/, gui_learning/, meeting_observation/, self_improvement/, and workflow_learning/. One of the most substantive nodes in the area.

psyche-orchestrator#

The central session-coordination service (services/psyche/orchestrator, ~11K LOC). src/orchestrator/session/ owns the lifecycle (types.py, lifecycle.py with session factory/timeout/handoff, state.py for checkpoints/persistence); resources/ allocates GPU/CPU/memory with pooling and scaling; pipeline/ routes real-time audio/video/data streams; multiparty/ manages participants, turns, and synchronization; caching/ provides vector-similarity semantic LLM caching; and batching/ does request batching. The tests/test_session_state_store_resolution.py exercises the state-store wiring. This is the conductor that drives the media engines per session.

psyche-perception-engine#

The computer-vision service (services/psyche/perception-engine, gpu:optional, ~8.6K LOC). src/perception_engine/emotion/ contains the real detection stack — face_detector.py, landmark_detector.py, emotion_classifier.py, audio_emotion.py, and multi-modal fusion.py — while screen/ does OCR / UI-element / document analysis, intent/ classifies speech and behavioural intent, and context/ performs temporal multi-modal aggregation. The README documents multiple face-detection backends (RetinaFace/MTCNN/MediaPipe/ OpenCV) and OCR engines (Tesseract/EasyOCR) behind these modules.

psyche-persona-service#

The identity/personality service (services/psyche/persona-service, feature:persona, ~8.5K LOC), structured as two packages. src/persona_service/ is the thin FastAPI surface (main.py); src/persona_system/ is the real business logic, split into identity/, personality/ (Big-Five model), expertise/ (domains and knowledge boundaries), memory/ (facts/opinions/story), constraints/ (response validation), drift/ (drift detection), and assets/ (avatar/voice profile creation) — each with its own manager.py/types.py. The PersonaSystem unified interface composes these into the persona a session runs as.

psyche-security#

The cross-cutting security/privacy library (services/psyche/security, platform:backend, ~5.4K LOC). src/security/ is organised by concern: auth/ (authentication.py, authorization.py — OAuth2/JWT/RBAC), data_protection/ (encryption, PII detection, retention), ai_security/ (AI-specific safeguards), privacy/ and compliance/ (regulatory controls), and monitoring/ (security telemetry). It is the safety substrate the other Psyche services import rather than a standalone application.

psyche-tavus-integration#

The external SOTA-avatar provider integration (services/psyche/tavus-integration, platform:backend, ~8K LOC). src/tavus_integration/api/ is a real Tavus REST client (client.py, auth.py, webhooks.py); replica/manager.py handles digital-twin creation from training-video upload with status monitoring and quality validation; and persona/manager.py builds system-prompt/guardrail persona configs and associates them with replicas. The README maps this onto Tavus's Phoenix-4 / Sparrow-1 / Raven-1 models — a genuine integration that depends on the Tavus API and credentials, offering an alternative to the in-house psyche-avatar-engine path.

psyche-tool-framework#

The tool-orchestration framework with MCP support (services/psyche/tool-framework, ~11K LOC, FastAPI main.py). src/tool_framework/registry/ defines tools with OpenAPI-compatible schemas (types.py, registry.py, validator.py); router/ executes them with retries, an LRU cache.py, a circuit breaker, and a tool-call parser.py; mcp/ implements a Model Context Protocol server (server.py, security.py, adapters.py); and mock_tools/ provides development stand-ins. tests/test_knowledge_base_mcp_adapter.py exercises the adapter wiring to psyche-knowledge-base. This is the layer that gives the assistant callable tools.

psyche-video-conferencing#

A library-packaged duplicate of psyche-conferencing. Its project.json is projectType: "library" (platform:backend), and its src/video_conferencing/ tree is byte-for-byte identical to the conferencing service's except for the top-level __init__.py docstring — the same gateway/, zoom/, teams/, meet/, webex/, webrtc/, virtual_devices/, meeting_intelligence/, etc. (~113 files, ~96K LOC). Notably its typecheck target is a literal true # python typecheck skipped, and it has no serve/docker targets. It is documented here as the library form of the conferencing code rather than a distinct implementation; the runnable application is psyche-conferencing.

psyche-voice-engine#

The real-time voice service (services/psyche/voice-engine, gpu:optional, ~57K LOC, very large test suite). src/voice_engine/tts/ carries real provider adapters — elevenlabs.py, openai_tts.py, cartesia.py, deepgram_aura.py, f5_tts.py, piper.py, orpheus.py — plus a provider_chain.py failover chain, streaming.py, voice_cloning.py, watermarking.py, and usage_audit.py. Alongside it sit stt/, phoneme/, viseme/ (MPEG-4 lip-sync), diarization/, turn_taking/, conversation/, audio/, failover/, sync/, captions/, commands/, transcript/, and accessibility/. The engine code is real; cloud TTS/STT providers require their credentials, and some paths target CUDA models.