The
services/psyche/area: a Poetry-workspace of Python/FastAPI services that make up the Psyche AI Virtual Assistant Platform — avatar rendering, voice, perception, computer use, conferencing, memory/learning, and the API edge that fronts them.
What this area is#
Psyche is the platform's embodied AI-assistant domain: the stack that turns
a persona into a real-time, audio-visual agent that can see, speak, render a
photoreal face, act on a desktop, and join a video call. Unlike most of the
Oshun monorepo (TypeScript/Nx), this area is Python 3.11+ on FastAPI,
managed as a single Poetry workspace. The root services/psyche/pyproject.toml
declares the shared dependency constraints and optional extras (ml, voice,
vision, avatar, aws, all) so each service installs only the heavy ML/GPU
stack it actually needs, and services/psyche/README.md records that the code
was migrated from the earlier "Serwaa" codebase.
Every node is a scope:psyche, layer:service, type:python Nx project. Most
are FastAPI applications with a uniform target set in their project.json
(serve via poetry run uvicorn, build/install via Poetry, lint with
ruff+mypy, test with pytest) and a Dockerfile; a few are tagged
runtime:fastapi, others platform:backend cross-cutting libraries. GPU
posture is encoded in tags too — gpu:required for the avatar engine,
gpu:optional for voice/perception/behavior.
The projects compose into a pipeline rather than a flat list. The
psyche-api-gateway is the HTTP/WebSocket edge; psyche-orchestrator owns
session lifecycle and resource allocation; the four "engines"
(avatar/voice/perception/behavior) are the real-time media plane;
psyche-persona-service, psyche-knowledge-base, and psyche-learning-system
provide identity, retrieval, and memory; psyche-tool-framework and
psyche-computer-use give the assistant the ability to call tools and operate a
desktop; psyche-conferencing/psyche-video-conferencing inject the avatar
into Zoom/Teams/Meet/Webex; and psyche-tavus-integration is an external
SOTA-avatar provider path. psyche-security and psyche-error-handling are the
cross-cutting resilience/safety libraries the rest depend on.
A note on honesty up front: the area is large and mostly substantively
implemented (avatar-engine alone is ~75K LOC of real PyTorch/3DGS code), but it
is not uniform. psyche-behavior-engine is an empty scaffold (its logic
lives inside the avatar engine), and psyche-video-conferencing is a
library-packaged duplicate of the psyche-conferencing source tree. Both are
flagged as such in their entries below rather than presented as independent
implementations. Many engines also depend on real external providers (Anthropic
Computer Use, Tavus, ElevenLabs/Deepgram/Cartesia, Voyage/OpenAI/Cohere) and
CUDA GPUs, so "implemented" here means the orchestration and provider-adapter
code is real, not that every path runs without creds or hardware.
How it fits the wider system#
Consumers reach Psyche through psyche-api-gateway (REST + WebSocket on port
8000), which is the only public surface; everything else is an internal service
the gateway and orchestrator coordinate. Within the area the dependency arrows
point toward the engines and the shared libraries: the orchestrator drives the
media engines, the engines lean on psyche-security for auth/data-protection
and psyche-error-handling for degradation/failover, and the persona/knowledge/
learning services supply the assistant's identity, facts, and long-term memory.
The boundary types Psyche exchanges with the rest of Oshun are defined elsewhere
— @psyche/contracts (libs/contracts/psyche) owns the validated wire schemas,
covered in the Contracts systems page — so this area focuses on behaviour and
runtime rather than on the contract surface. Outward, the heavy integrations are
with third-party platforms (conferencing SDKs, avatar/voice/LLM vendors) rather
than with sibling Oshun domains, which is why so many project.json files carry
platform/provider install-extras and GPU tags.
Entity reference#
psyche-api-gateway#
The REST + WebSocket edge for the whole platform
(services/psyche/api-gateway), a real FastAPI app built through the
create_app factory in src/api_gateway/app.py (registered middleware,
exception handlers, CORS/GZip, ORJSON). It carries a full router set under
src/api_gateway/routers/ — sessions, personas, knowledge, tools,
webhooks, websocket, experiments, features, and health
(liveness/readiness/dependency probes) — backed by SQLAlchemy models in
models/, a real-time layer in websocket/ (manager.py, handlers.py), a
webhook delivery service (services/webhooks.py, ~630 LOC), and DB migration
tooling (db/migrate.py). It is the unified front door the README documents at
:8000/docs, and the most integration-heavy node in the area.
psyche-avatar-engine#
The neural-rendering engine and the largest node here (~75K LOC across ~150
files, gpu:required). src/avatar_engine/engine.py defines AvatarEngine,
the orchestration class that imports torch, the GaussianModel/FLAMEModel,
and a real rasterization/rendering stack (rendering/rasterizer.py,
temporal_aa.py, ambient_occlusion.py, dynamic_resolution.py) for 3D
Gaussian Splatting avatars. Beyond rendering it spans nerf/, training/,
quantization/, lipsync/, expression/, eye/, body/, capture/,
culture/, validation/, video/, and a substantial accessibility/ module
(description_engine.py ~1.2K LOC, scene_analyzer.py ~0.8K LOC). It also
hosts the behavior/ submodule that the behavior-engine service nominally
fronts. Real implementation, but predicated on a CUDA GPU and ML extras.
psyche-behavior-engine#
An empty scaffold. The directory contains only README.md, Dockerfile,
project.json, pyproject.toml, and .gitkeep placeholders under src/ and
tests/ — there is no Python implementation. The README is explicit that "the
core behavioral logic is integrated into the Avatar Engine
(services/psyche/avatar-engine/src/avatar_engine/behavior/)," so this node is
a reserved FACS/expression API slot whose logic actually lives in
psyche-avatar-engine. It is documented as a placeholder, not a working
service.
psyche-computer-use#
The autonomous desktop-agent service (services/psyche/computer-use, ~23K LOC
plus a very large test suite of ~60 files). It implements an observe-reason-act
loop around Anthropic's Claude Computer Use API: src/computer_use/agent/ holds
loop.py, planning.py, dead_end.py, recovery.py, undo.py, and
explain.py, while environment/ provides the Xvfb-sandboxed desktop,
vision/ does OCR/UI-element detection, actions/ covers mouse/keyboard/file
operations, screen_sharing/ streams via WebRTC, and benchmark/ carries an
OSWorld-style evaluation harness (with baselines under benchmark_baselines/).
Real agent code, dependent on the Anthropic API and a Playwright/Xvfb runtime.
psyche-conferencing#
The live-meeting injection service (services/psyche/conferencing, FastAPI
application served on port 8006; its project.json docker-run target maps
/dev/video0 and it exposes install-zoom/install-teams/install-meet Nx
install targets). Its video_conferencing package (~113 files, ~96K LOC)
provides a platform-agnostic gateway/ adapter plus per-platform integrations
zoom/, teams/, meet/, webex/, webrtc/, real virtual_devices/
(v4l2loopback camera + PulseAudio injection), and multiparticipant/,
meeting_intelligence/, optimization/, perception/, and accessibility/
modules. This is the runnable conferencing application;
psyche-video-conferencing packages the same tree as a library (see below).
psyche-error-handling#
A cross-cutting resilience library (services/psyche/error-handling,
platform:backend, ~4K LOC, no FastAPI runtime). It is real and self-contained:
src/error_handling/errors/ defines a typed exception hierarchy (types.py,
~584 LOC) and automatic classification (detection.py, ~802 LOC);
degradation/__init__.py (~663 LOC) implements multi-level graceful degradation
with component-specific degraders; recovery/__init__.py (~858 LOC) carries
retry policies, the circuit-breaker pattern, and multi-provider failover; and
communication/__init__.py (~678 LOC) handles user-facing error messaging. A
genuine implementation other Psyche services import for fault tolerance.
psyche-knowledge-base#
The RAG and document-processing service (services/psyche/knowledge-base,
feature:rag, ~14K LOC). src/knowledge_base/ is organised by pipeline stage:
ingestion/ (parsers, chunking strategies, cleaning), embedding/,
retrieval/, management/, and rag/ (pipeline.py, generator.py,
evaluator.py, prompts.py) implementing the retrieve-rank-generate flow with
quality evaluation. The README documents multiple embedding providers
(Voyage/OpenAI/Cohere/BGE) and vector stores
(Pinecone/Qdrant/pgvector/in-memory) behind these adapters; the orchestration is
real, individual providers require their respective credentials.
psyche-learning-system#
The memory and skill-acquisition service (services/psyche/learning-system,
~72K LOC, large test suite, main.py FastAPI app). It implements a
MemGPT/Letta- style hierarchical memory:
src/learning_system/persistent_memory/ holds memory_tools.py,
decay_consolidation.py, context_management.py, and persistence.py,
complemented by skill_library/ (Voyager-style skill discovery/ composition),
knowledge_graph/ (temporal entity-relationship memory), procedural_memory/,
instruction_memory/, verbal_instruction/, learning_from_demonstration/,
experience_based_learning/, gui_learning/, meeting_observation/,
self_improvement/, and workflow_learning/. One of the most substantive nodes
in the area.
psyche-orchestrator#
The central session-coordination service (services/psyche/orchestrator, ~11K
LOC). src/orchestrator/session/ owns the lifecycle (types.py, lifecycle.py
with session factory/timeout/handoff, state.py for checkpoints/persistence);
resources/ allocates GPU/CPU/memory with pooling and scaling; pipeline/
routes real-time audio/video/data streams; multiparty/ manages participants,
turns, and synchronization; caching/ provides vector-similarity semantic LLM
caching; and batching/ does request batching. The
tests/test_session_state_store_resolution.py exercises the state-store wiring.
This is the conductor that drives the media engines per session.
psyche-perception-engine#
The computer-vision service (services/psyche/perception-engine,
gpu:optional, ~8.6K LOC). src/perception_engine/emotion/ contains the real
detection stack — face_detector.py, landmark_detector.py,
emotion_classifier.py, audio_emotion.py, and multi-modal fusion.py — while
screen/ does OCR / UI-element / document analysis, intent/ classifies speech
and behavioural intent, and context/ performs temporal multi-modal
aggregation. The README documents multiple face-detection backends
(RetinaFace/MTCNN/MediaPipe/ OpenCV) and OCR engines (Tesseract/EasyOCR) behind
these modules.
psyche-persona-service#
The identity/personality service (services/psyche/persona-service,
feature:persona, ~8.5K LOC), structured as two packages.
src/persona_service/ is the thin FastAPI surface (main.py);
src/persona_system/ is the real business logic, split into identity/,
personality/ (Big-Five model), expertise/ (domains and knowledge
boundaries), memory/ (facts/opinions/story), constraints/ (response
validation), drift/ (drift detection), and assets/ (avatar/voice profile
creation) — each with its own manager.py/types.py. The PersonaSystem
unified interface composes these into the persona a session runs as.
psyche-security#
The cross-cutting security/privacy library (services/psyche/security,
platform:backend, ~5.4K LOC). src/security/ is organised by concern: auth/
(authentication.py, authorization.py — OAuth2/JWT/RBAC), data_protection/
(encryption, PII detection, retention), ai_security/ (AI-specific safeguards),
privacy/ and compliance/ (regulatory controls), and monitoring/ (security
telemetry). It is the safety substrate the other Psyche services import rather
than a standalone application.
psyche-tavus-integration#
The external SOTA-avatar provider integration
(services/psyche/tavus-integration, platform:backend, ~8K LOC).
src/tavus_integration/api/ is a real Tavus REST client (client.py,
auth.py, webhooks.py); replica/manager.py handles digital-twin creation
from training-video upload with status monitoring and quality validation; and
persona/manager.py builds system-prompt/guardrail persona configs and
associates them with replicas. The README maps this onto Tavus's Phoenix-4 /
Sparrow-1 / Raven-1 models — a genuine integration that depends on the Tavus API
and credentials, offering an alternative to the in-house psyche-avatar-engine
path.
psyche-tool-framework#
The tool-orchestration framework with MCP support
(services/psyche/tool-framework, ~11K LOC, FastAPI main.py).
src/tool_framework/registry/ defines tools with OpenAPI-compatible schemas
(types.py, registry.py, validator.py); router/ executes them with
retries, an LRU cache.py, a circuit breaker, and a tool-call parser.py;
mcp/ implements a Model Context Protocol server (server.py, security.py,
adapters.py); and mock_tools/ provides development stand-ins.
tests/test_knowledge_base_mcp_adapter.py exercises the adapter wiring to
psyche-knowledge-base. This is the layer that gives the assistant callable
tools.
psyche-video-conferencing#
A library-packaged duplicate of psyche-conferencing. Its project.json is
projectType: "library" (platform:backend), and its src/video_conferencing/
tree is byte-for-byte identical to the conferencing service's except for the
top-level __init__.py docstring — the same gateway/, zoom/, teams/,
meet/, webex/, webrtc/, virtual_devices/, meeting_intelligence/, etc.
(~113 files, ~96K LOC). Notably its typecheck target is a literal
true # python typecheck skipped, and it has no serve/docker targets. It is
documented here as the library form of the conferencing code rather than a
distinct implementation; the runnable application is psyche-conferencing.
psyche-voice-engine#
The real-time voice service (services/psyche/voice-engine, gpu:optional,
~57K LOC, very large test suite). src/voice_engine/tts/ carries real provider
adapters — elevenlabs.py, openai_tts.py, cartesia.py, deepgram_aura.py,
f5_tts.py, piper.py, orpheus.py — plus a provider_chain.py failover
chain, streaming.py, voice_cloning.py, watermarking.py, and
usage_audit.py. Alongside it sit stt/, phoneme/, viseme/ (MPEG-4
lip-sync), diarization/, turn_taking/, conversation/, audio/,
failover/, sync/, captions/, commands/, transcript/, and
accessibility/. The engine code is real; cloud TTS/STT providers require their
credentials, and some paths target CUDA models.