# Psyche — Systems Deep Dive

> The `services/psyche/` area: a Poetry-workspace of Python/FastAPI services
> that make up the **Psyche AI Virtual Assistant Platform** — avatar rendering,
> voice, perception, computer use, conferencing, memory/learning, and the API
> edge that fronts them.

## What this area is

Psyche is the platform's **embodied AI-assistant domain**: the stack that turns
a persona into a real-time, audio-visual agent that can see, speak, render a
photoreal face, act on a desktop, and join a video call. Unlike most of the
Oshun monorepo (TypeScript/Nx), this area is **Python 3.11+ on FastAPI**,
managed as a single Poetry workspace. The root `services/psyche/pyproject.toml`
declares the shared dependency constraints and optional extras (`ml`, `voice`,
`vision`, `avatar`, `aws`, `all`) so each service installs only the heavy ML/GPU
stack it actually needs, and `services/psyche/README.md` records that the code
was migrated from the earlier "Serwaa" codebase.

Every node is a `scope:psyche`, `layer:service`, `type:python` Nx project. Most
are FastAPI `application`s with a uniform target set in their `project.json`
(`serve` via `poetry run uvicorn`, `build`/`install` via Poetry, `lint` with
ruff+mypy, `test` with pytest) and a `Dockerfile`; a few are tagged
`runtime:fastapi`, others `platform:backend` cross-cutting libraries. GPU
posture is encoded in tags too — `gpu:required` for the avatar engine,
`gpu:optional` for voice/perception/behavior.

The projects compose into a pipeline rather than a flat list. The
`psyche-api-gateway` is the HTTP/WebSocket edge; `psyche-orchestrator` owns
session lifecycle and resource allocation; the four "engines"
(`avatar`/`voice`/`perception`/`behavior`) are the real-time media plane;
`psyche-persona-service`, `psyche-knowledge-base`, and `psyche-learning-system`
provide identity, retrieval, and memory; `psyche-tool-framework` and
`psyche-computer-use` give the assistant the ability to call tools and operate a
desktop; `psyche-conferencing`/`psyche-video-conferencing` inject the avatar
into Zoom/Teams/Meet/Webex; and `psyche-tavus-integration` is an external
SOTA-avatar provider path. `psyche-security` and `psyche-error-handling` are the
cross-cutting resilience/safety libraries the rest depend on.

A note on honesty up front: the area is large and mostly substantively
implemented (avatar-engine alone is ~75K LOC of real PyTorch/3DGS code), but it
is not uniform. **`psyche-behavior-engine` is an empty scaffold** (its logic
lives inside the avatar engine), and **`psyche-video-conferencing` is a
library-packaged duplicate** of the `psyche-conferencing` source tree. Both are
flagged as such in their entries below rather than presented as independent
implementations. Many engines also depend on real external providers (Anthropic
Computer Use, Tavus, ElevenLabs/Deepgram/Cartesia, Voyage/OpenAI/Cohere) and
CUDA GPUs, so "implemented" here means the orchestration and provider-adapter
code is real, not that every path runs without creds or hardware.

## How it fits the wider system

Consumers reach Psyche through `psyche-api-gateway` (REST + WebSocket on port
8000), which is the only public surface; everything else is an internal service
the gateway and orchestrator coordinate. Within the area the dependency arrows
point toward the engines and the shared libraries: the orchestrator drives the
media engines, the engines lean on `psyche-security` for auth/data-protection
and `psyche-error-handling` for degradation/failover, and the persona/knowledge/
learning services supply the assistant's identity, facts, and long-term memory.

The boundary types Psyche exchanges with the rest of Oshun are defined elsewhere
— `@psyche/contracts` (`libs/contracts/psyche`) owns the validated wire schemas,
covered in the Contracts systems page — so this area focuses on behaviour and
runtime rather than on the contract surface. Outward, the heavy integrations are
with third-party platforms (conferencing SDKs, avatar/voice/LLM vendors) rather
than with sibling Oshun domains, which is why so many `project.json` files carry
platform/provider install-extras and GPU tags.

## Entity reference

### psyche-api-gateway

The REST + WebSocket edge for the whole platform
(`services/psyche/api-gateway`), a real FastAPI app built through the
`create_app` factory in `src/api_gateway/app.py` (registered middleware,
exception handlers, CORS/GZip, ORJSON). It carries a full router set under
`src/api_gateway/routers/` — `sessions`, `personas`, `knowledge`, `tools`,
`webhooks`, `websocket`, `experiments`, `features`, and `health`
(liveness/readiness/dependency probes) — backed by SQLAlchemy models in
`models/`, a real-time layer in `websocket/` (`manager.py`, `handlers.py`), a
webhook delivery service (`services/webhooks.py`, ~630 LOC), and DB migration
tooling (`db/migrate.py`). It is the unified front door the README documents at
`:8000/docs`, and the most integration-heavy node in the area.

### psyche-avatar-engine

The neural-rendering engine and the largest node here (~75K LOC across ~150
files, `gpu:required`). `src/avatar_engine/engine.py` defines `AvatarEngine`,
the orchestration class that imports `torch`, the `GaussianModel`/`FLAMEModel`,
and a real rasterization/rendering stack (`rendering/rasterizer.py`,
`temporal_aa.py`, `ambient_occlusion.py`, `dynamic_resolution.py`) for 3D
Gaussian Splatting avatars. Beyond rendering it spans `nerf/`, `training/`,
`quantization/`, `lipsync/`, `expression/`, `eye/`, `body/`, `capture/`,
`culture/`, `validation/`, `video/`, and a substantial `accessibility/` module
(`description_engine.py` ~1.2K LOC, `scene_analyzer.py` ~0.8K LOC). It also
hosts the `behavior/` submodule that the behavior-engine service nominally
fronts. Real implementation, but predicated on a CUDA GPU and ML extras.

### psyche-behavior-engine

**An empty scaffold.** The directory contains only `README.md`, `Dockerfile`,
`project.json`, `pyproject.toml`, and `.gitkeep` placeholders under `src/` and
`tests/` — there is no Python implementation. The README is explicit that "the
core behavioral logic is integrated into the Avatar Engine
(`services/psyche/avatar-engine/src/avatar_engine/behavior/`)," so this node is
a reserved FACS/expression API slot whose logic actually lives in
`psyche-avatar-engine`. It is documented as a placeholder, not a working
service.

### psyche-computer-use

The autonomous desktop-agent service (`services/psyche/computer-use`, ~23K LOC
plus a very large test suite of ~60 files). It implements an observe-reason-act
loop around Anthropic's Claude Computer Use API: `src/computer_use/agent/` holds
`loop.py`, `planning.py`, `dead_end.py`, `recovery.py`, `undo.py`, and
`explain.py`, while `environment/` provides the Xvfb-sandboxed desktop,
`vision/` does OCR/UI-element detection, `actions/` covers mouse/keyboard/file
operations, `screen_sharing/` streams via WebRTC, and `benchmark/` carries an
OSWorld-style evaluation harness (with baselines under `benchmark_baselines/`).
Real agent code, dependent on the Anthropic API and a Playwright/Xvfb runtime.

### psyche-conferencing

The live-meeting injection service (`services/psyche/conferencing`, FastAPI
`application` served on port 8006; its `project.json` `docker-run` target maps
`/dev/video0` and it exposes `install-zoom`/`install-teams`/`install-meet` Nx
install targets). Its `video_conferencing` package (~113 files, ~96K LOC)
provides a platform-agnostic `gateway/` adapter plus per-platform integrations
`zoom/`, `teams/`, `meet/`, `webex/`, `webrtc/`, real `virtual_devices/`
(v4l2loopback camera + PulseAudio injection), and `multiparticipant/`,
`meeting_intelligence/`, `optimization/`, `perception/`, and `accessibility/`
modules. This is the runnable conferencing application;
`psyche-video-conferencing` packages the same tree as a library (see below).

### psyche-error-handling

A cross-cutting resilience library (`services/psyche/error-handling`,
`platform:backend`, ~4K LOC, no FastAPI runtime). It is real and self-contained:
`src/error_handling/errors/` defines a typed exception hierarchy (`types.py`,
~584 LOC) and automatic classification (`detection.py`, ~802 LOC);
`degradation/__init__.py` (~663 LOC) implements multi-level graceful degradation
with component-specific degraders; `recovery/__init__.py` (~858 LOC) carries
retry policies, the circuit-breaker pattern, and multi-provider failover; and
`communication/__init__.py` (~678 LOC) handles user-facing error messaging. A
genuine implementation other Psyche services import for fault tolerance.

### psyche-knowledge-base

The RAG and document-processing service (`services/psyche/knowledge-base`,
`feature:rag`, ~14K LOC). `src/knowledge_base/` is organised by pipeline stage:
`ingestion/` (parsers, chunking strategies, cleaning), `embedding/`,
`retrieval/`, `management/`, and `rag/` (`pipeline.py`, `generator.py`,
`evaluator.py`, `prompts.py`) implementing the retrieve-rank-generate flow with
quality evaluation. The README documents multiple embedding providers
(Voyage/OpenAI/Cohere/BGE) and vector stores
(Pinecone/Qdrant/pgvector/in-memory) behind these adapters; the orchestration is
real, individual providers require their respective credentials.

### psyche-learning-system

The memory and skill-acquisition service (`services/psyche/learning-system`,
~72K LOC, large test suite, `main.py` FastAPI app). It implements a
MemGPT/Letta- style hierarchical memory:
`src/learning_system/persistent_memory/` holds `memory_tools.py`,
`decay_consolidation.py`, `context_management.py`, and `persistence.py`,
complemented by `skill_library/` (Voyager-style skill discovery/ composition),
`knowledge_graph/` (temporal entity-relationship memory), `procedural_memory/`,
`instruction_memory/`, `verbal_instruction/`, `learning_from_demonstration/`,
`experience_based_learning/`, `gui_learning/`, `meeting_observation/`,
`self_improvement/`, and `workflow_learning/`. One of the most substantive nodes
in the area.

### psyche-orchestrator

The central session-coordination service (`services/psyche/orchestrator`, ~11K
LOC). `src/orchestrator/session/` owns the lifecycle (`types.py`, `lifecycle.py`
with session factory/timeout/handoff, `state.py` for checkpoints/persistence);
`resources/` allocates GPU/CPU/memory with pooling and scaling; `pipeline/`
routes real-time audio/video/data streams; `multiparty/` manages participants,
turns, and synchronization; `caching/` provides vector-similarity semantic LLM
caching; and `batching/` does request batching. The
`tests/test_session_state_store_resolution.py` exercises the state-store wiring.
This is the conductor that drives the media engines per session.

### psyche-perception-engine

The computer-vision service (`services/psyche/perception-engine`,
`gpu:optional`, ~8.6K LOC). `src/perception_engine/emotion/` contains the real
detection stack — `face_detector.py`, `landmark_detector.py`,
`emotion_classifier.py`, `audio_emotion.py`, and multi-modal `fusion.py` — while
`screen/` does OCR / UI-element / document analysis, `intent/` classifies speech
and behavioural intent, and `context/` performs temporal multi-modal
aggregation. The README documents multiple face-detection backends
(RetinaFace/MTCNN/MediaPipe/ OpenCV) and OCR engines (Tesseract/EasyOCR) behind
these modules.

### psyche-persona-service

The identity/personality service (`services/psyche/persona-service`,
`feature:persona`, ~8.5K LOC), structured as two packages.
`src/persona_service/` is the thin FastAPI surface (`main.py`);
`src/persona_system/` is the real business logic, split into `identity/`,
`personality/` (Big-Five model), `expertise/` (domains and knowledge
boundaries), `memory/` (facts/opinions/story), `constraints/` (response
validation), `drift/` (drift detection), and `assets/` (avatar/voice profile
creation) — each with its own `manager.py`/`types.py`. The `PersonaSystem`
unified interface composes these into the persona a session runs as.

### psyche-security

The cross-cutting security/privacy library (`services/psyche/security`,
`platform:backend`, ~5.4K LOC). `src/security/` is organised by concern: `auth/`
(`authentication.py`, `authorization.py` — OAuth2/JWT/RBAC), `data_protection/`
(encryption, PII detection, retention), `ai_security/` (AI-specific safeguards),
`privacy/` and `compliance/` (regulatory controls), and `monitoring/` (security
telemetry). It is the safety substrate the other Psyche services import rather
than a standalone application.

### psyche-tavus-integration

The external SOTA-avatar provider integration
(`services/psyche/tavus-integration`, `platform:backend`, ~8K LOC).
`src/tavus_integration/api/` is a real Tavus REST client (`client.py`,
`auth.py`, `webhooks.py`); `replica/manager.py` handles digital-twin creation
from training-video upload with status monitoring and quality validation; and
`persona/manager.py` builds system-prompt/guardrail persona configs and
associates them with replicas. The README maps this onto Tavus's Phoenix-4 /
Sparrow-1 / Raven-1 models — a genuine integration that depends on the Tavus API
and credentials, offering an alternative to the in-house `psyche-avatar-engine`
path.

### psyche-tool-framework

The tool-orchestration framework with MCP support
(`services/psyche/tool-framework`, ~11K LOC, FastAPI `main.py`).
`src/tool_framework/registry/` defines tools with OpenAPI-compatible schemas
(`types.py`, `registry.py`, `validator.py`); `router/` executes them with
retries, an LRU `cache.py`, a circuit breaker, and a tool-call `parser.py`;
`mcp/` implements a Model Context Protocol server (`server.py`, `security.py`,
`adapters.py`); and `mock_tools/` provides development stand-ins.
`tests/test_knowledge_base_mcp_adapter.py` exercises the adapter wiring to
`psyche-knowledge-base`. This is the layer that gives the assistant callable
tools.

### psyche-video-conferencing

A **library-packaged duplicate** of `psyche-conferencing`. Its `project.json` is
`projectType: "library"` (`platform:backend`), and its `src/video_conferencing/`
tree is byte-for-byte identical to the conferencing service's except for the
top-level `__init__.py` docstring — the same `gateway/`, `zoom/`, `teams/`,
`meet/`, `webex/`, `webrtc/`, `virtual_devices/`, `meeting_intelligence/`, etc.
(~113 files, ~96K LOC). Notably its `typecheck` target is a literal
`true # python typecheck skipped`, and it has no `serve`/`docker` targets. It is
documented here as the library form of the conferencing code rather than a
distinct implementation; the runnable application is `psyche-conferencing`.

### psyche-voice-engine

The real-time voice service (`services/psyche/voice-engine`, `gpu:optional`,
~57K LOC, very large test suite). `src/voice_engine/tts/` carries real provider
adapters — `elevenlabs.py`, `openai_tts.py`, `cartesia.py`, `deepgram_aura.py`,
`f5_tts.py`, `piper.py`, `orpheus.py` — plus a `provider_chain.py` failover
chain, `streaming.py`, `voice_cloning.py`, `watermarking.py`, and
`usage_audit.py`. Alongside it sit `stt/`, `phoneme/`, `viseme/` (MPEG-4
lip-sync), `diarization/`, `turn_taking/`, `conversation/`, `audio/`,
`failover/`, `sync/`, `captions/`, `commands/`, `transcript/`, and
`accessibility/`. The engine code is real; cloud TTS/STT providers require their
credentials, and some paths target CUDA models.
