Psyche Domain — Technical Specifications

Technical specification for the Psyche Hyper-Realistic AI Virtual Assistant Platform. This document describes what is implemented in code: the API Gateway request/response models, enums, the session-orchestration state machine, WebSocket protocol, service topology, configuration, and persistence bootstrap.

Grounding scope. Every schema, field, enum value, endpoint, and event below is taken from source under services/psyche/, apps/psyche/admin/, and libs/contracts/psyche/. Where a concept is named in code only as a directory or interface without a concrete persisted schema, it is described at that level and not embellished. The API Gateway currently keeps all resources in per-process in-memory dictionaries — there is no relational persistence of personas/sessions/knowledge/tools/webhooks yet (see §10).

This document is the authoritative reference for engineers working on Psyche services. It maps exactly to what exists in source: model fields, enum values, endpoint paths, and configuration keys. Where the current implementation diverges from the intended design (for example, in-memory storage instead of a real database), those gaps are called out explicitly in notes. Use this alongside architecture.md (system topology and ML pipelines) and features.md (user-facing capabilities).

1. Implementation Status#

Psyche is implemented as a polyglot domain:

• Python services — 16 services under services/psyche/, each a Poetry package with pyproject.toml, project.json, src/, and tests/. The workspace root services/psyche/pyproject.toml declares every service as a path dependency. The framework is FastAPI on Python 3.11+.
• TypeScript libraries — 135 library directories under libs/psyche/ (predominantly TypeScript: ~1,690 .ts files vs ~130 .py files), each with package.json, project.json, tsconfig.json, tsup.config.ts, and vitest.config.ts.
• Admin web app — apps/psyche/admin, a Next.js application.
• Contracts — libs/contracts/psyche (@psyche/contracts), a Zod-based shared-types package. Only the common module is implemented; per-domain contract modules are declared in package.json exports but their files do not yet exist (see §18).

The branding Serwaa is the former codename and still appears in API titles, default config values, MinIO/storage prefixes, and webhook examples.

2. Code Layout#

The source tree below shows every deployable service and the admin application. Note that two service directories (conferencing/, persona-service/) ship Python package names that differ from their directory names — this is called out explicitly below the tree.

services/psyche/
  api-gateway/        FastAPI REST + WebSocket gateway (psyche-api-gateway)
  orchestrator/       Session lifecycle, pipeline routing, resources (psyche-orchestrator)
  persona-service/    Persona identity/personality/expertise/memory (psyche-persona-service)
  avatar-engine/      3DGS / NeRF rendering, FLAME, training (psyche-avatar-engine)
  voice-engine/       TTS, STT, diarization, visemes, turn-taking (psyche-voice-engine)
  behavior-engine/    (psyche-behavior-engine) — service shell, src/ empty
  perception-engine/  Face/emotion/intent/screen perception (psyche-perception-engine)
  computer-use/       Browser/desktop automation agent (psyche-computer-use)
  conferencing/       Zoom/Teams/Meet/Webex/WebRTC adapters (psyche-conferencing)
  video-conferencing/ (psyche-video-conferencing)
  knowledge-base/     Ingestion, embedding, RAG, retrieval (psyche-knowledge-base)
  learning-system/    (psyche-learning-system)
  tool-framework/     Tool registry, router, MCP server (psyche-tool-framework)
  tavus-integration/  (psyche-tavus-integration)
  security/           (psyche-security)
  error-handling/     (psyche-error-handling)

apps/psyche/admin/    Next.js admin dashboard
libs/psyche/*         135 TypeScript libraries
libs/contracts/psyche @psyche/contracts (Zod)

The Python package name inside each service does not always match the directory name: conferencing/ ships the video_conferencing package, and persona-service/ ships both a persona_service package (FastAPI app) and a persona_system package (the domain logic).

3. API Gateway Data Models#

All API Gateway request/response models are Pydantic v2 models. They live in services/psyche/api-gateway/src/api_gateway/models/. Every model inherits from BaseSchema, which sets strict=True, from_attributes=True, validate_assignment=True, use_enum_values=True, populate_by_name=True, and extra="forbid". Response models additionally mix in TimestampMixin (created_at, updated_at).

3.1 Persona Models (models/personas.py)#

PersonalityConfig#

PersonalityConfig is the API Gateway's flat personality representation. It is not the OCEAN Big Five model — that richer model lives in the Persona Service (see §9). The API uses a simpler six-trait flat set for request/response serialization.

VoiceConfig#

VoiceConfig specifies the TTS provider and voice parameters attached to a persona.

AvatarConfig#

AvatarConfig links a persona to a trained avatar model and controls the visual presentation.

PersonaCreate (request)#

PersonaCreate is the body for POST /api/v1/personas. Every field except name has a default.

PersonaUpdate is the same field set with every field optional, plus is_active: bool | null. There is no personality/voice/avatar-equivalent of the OCEAN model, no jobTitle, company, shortBio, topicBoundaries, tools, or profilePhotoUrl field on these API models.

PersonaResponse#

PersonaResponse is the shape returned by all persona read endpoints. It adds identity and lifecycle fields on top of the PersonaCreate fields.

Adds id (string, format persona_NNNNNNNN), organization_id, is_active: bool, training_status: str (default "not_started"), plus created_at/updated_at. The full PersonalityConfig, VoiceConfig, and AvatarConfig are embedded.

3.2 Session Models (models/sessions.py)#

SessionCreate (request)#

SessionCreate is the body for POST /api/v1/sessions. The platform field can be omitted — it is auto-detected from the meeting URL when possible.

JoinMeetingRequest carries a required meeting_url, optional platform, and optional display_name (maxlen 100).

SessionResponse#

SessionResponse is the shape returned for all session read operations. The duration_seconds field is computed when the session ends.

SessionMetrics aggregates total_sessions, active_sessions, avg_duration_seconds, total_messages. SessionListFilters defines query filters by status, persona_id, platform, started_after, started_before.

3.3 Knowledge Models (models/knowledge.py)#

KnowledgeDocumentCreate represents a document being added to the knowledge base — either with inline content or a source_url to fetch from.

KnowledgeDocumentCreate: title (1–200), content (str | null, maxlen 1,000,000), source_url (str | null, maxlen 500, HTTP(S)-validated), document_type (DocumentType, default text), persona_ids: list[str], tags: list[str] (maxlen 20, normalized), metadata.

KnowledgeDocumentResponse: adds id (doc_NNNNNNNN), organization_id, status (DocumentStatus), chunk_count, token_count, size_bytes, error_message, plus timestamps.

KnowledgeSearchRequest: query (1–1000), optional persona_id, optional tags, top_k (1–20, default 5). KnowledgeSearchResult: document_id, document_title, chunk_id, content, score (0–1), metadata. KnowledgeSearchResponse: query, results, total_results.

The gateway's search handler currently performs case-insensitive title matching with a fixed score of 0.85 and does not query a vector store. The Qdrant-backed semantic path is implemented in services/psyche/knowledge-base (see §15) but is not wired into the gateway endpoint.

3.4 Tool Models (models/tools.py)#

Tools define callable integrations that personas can invoke mid-conversation. The ToolParameter model describes a single typed input parameter; ToolCreate assembles those parameters into a registered tool definition.

ToolParameter: name (1–100), type (ParameterType), description (maxlen 500), required: bool, default: Any, enum: list[str] | null, items: dict | null, properties: dict | null.

ToolCreate: name (1–100, must be snake_case, not digit-leading, lower-cased), description (1–1000), tool_type (ToolType, default function), parameters: list[ToolParameter], endpoint_url / mcp_server_url (str | null, maxlen 500, HTTP(S)-validated), auth_type (str | null), persona_ids, timeout_seconds (1–300, default 30), retry_config: dict, metadata.

ToolResponse: adds id (tool_NNNNNNNN), organization_id, is_active: bool, execution_count, success_rate (0–1, default 1.0), avg_latency_ms, plus timestamps.

ToolTestRequest/ToolTestResponse model the /test endpoint: success: bool, result: Any, error: str | null, latency_ms.

3.5 Webhook Models (models/webhooks.py)#

Webhooks enable external systems to receive HTTP callbacks when Psyche events occur. The WebhookCreate model enforces HTTPS for security. A signing secret is auto-generated if not provided; only the first 10 characters are returned in responses to avoid exposing the full secret.

WebhookCreate: url (maxlen 500, must be HTTPS), events: list[WebhookEvent] (minlen 1), description (maxlen 500), secret: str | null (maxlen 200, auto-generated as whsec_<token> if absent), headers: dict[str, str], enabled: bool, metadata.

WebhookResponse: id (webhook_NNNNNNNN), organization_id, url, events, description, status (WebhookStatus), headers, secret_prefix (first 10 chars of the signing secret), delivery_stats: WebhookDeliveryStats, metadata, timestamps.

WebhookDeliveryStats: total_deliveries, successful_deliveries, failed_deliveries, success_rate (0–1), avg_latency_ms, last_delivery_at, last_success_at, last_failure_at.

WebhookDelivery: id, webhook_id, event_type, status_code: int | null, success: bool, attempts (>= 1), latency_ms, error: str | null, request_headers, response_body: str | null, created_at.

3.6 Base / Envelope Models (models/base.py)#

These generic envelope models wrap all API responses for consistency. Every success response is either SuccessResponse[T] (single item) or PaginatedResponse[T] (list); errors always follow ErrorResponse.

• SuccessResponse[T] — { data: T, meta: dict }.
• PaginatedResponse[T] — { data: list[T], meta: PaginationMeta }.
• PaginationMeta — total, page, per_page (1–100), total_pages, has_next, has_prev.
• ErrorResponse — { error: ErrorDetail } where ErrorDetail is { code, message, details }.
• DeleteResponse — { deleted: bool, id: str }.

4. Enumerations#

All enums are string enums unless noted. Sources are cited per enum. A key thing to be aware of: the SessionStatus enum is defined independently in three places — the API Gateway, the Orchestrator, and the shared Contracts package — and the three sets of values do not match. This is an intentional layering decision (each service models its own lifecycle state), but it means you must be careful which type you are using in any given context.

API Gateway#

These enums are defined in the API Gateway models and control the values accepted in REST request bodies and returned in responses.

WebhookEvent values: session.started, session.ended, session.error, conversation.message, conversation.turn_started, conversation.turn_ended, meeting.joined, meeting.left, meeting.participant_joined, meeting.participant_left, tool.executed, tool.error, error.occurred, persona.updated.

WebSocket (api_gateway/websocket/models.py)#

These enums govern the real-time WebSocket protocol. WSMessageType separates the client-to-server and server-to-client message shapes; WSEventType is the set of session-level events clients can subscribe to; CommandType is the set of control commands clients can issue.

Orchestrator (orchestrator/session/types.py)#

The Orchestrator has its own richer SessionStatus enum that covers internal lifecycle states (initializing, resuming, terminating) not exposed at the API layer. The ConversationPhase enum tracks where in the conversation arc the session currently sits.

The orchestrator's SessionStatus is a distinct enum from the API Gateway's. The two services model session lifecycle independently and are not currently wired together.

Persona Service (persona-service/src/persona_system/)#

The Persona Service has the richest set of enums in the domain, covering every dimension of persona identity, behavior, memory, and asset lifecycle. They are grouped below by their source module within persona_system.

Voice Engine (voice-engine/src/voice_engine/)#

The Voice Engine defines separate ProviderType enums for TTS and STT — they are defined in different modules and have different values.

Perception Engine (perception-engine/src/perception_engine/emotion/types.py)#

These enums model the emotional and attentional signals extracted from participant video and audio. EmotionArousal and EmotionValence provide a two-axis model (positive/negative, high/low activation) that complements the discrete EmotionType classification.

Avatar Engine (avatar-engine/src/avatar_engine/config.py)#

Computer-Use (computer-use/src/computer_use/actions/types.py)#

ActionType is an auto-valued enum covering every action the computer-use agent can take. It is organized into logical groups: mouse, keyboard, browser, file, and utility operations. ScrollDirection and MouseButton are small enums used as parameters on mouse actions.

ActionType (auto-valued enum) covers mouse actions (SCREENSHOT, LEFT_CLICK, RIGHT_CLICK, MIDDLE_CLICK, DOUBLE_CLICK, TRIPLE_CLICK, MOUSE_MOVE, LEFT_CLICK_DRAG, LEFT_MOUSE_DOWN, LEFT_MOUSE_UP, SCROLL), keyboard actions (TYPE, KEY, HOLD_KEY, CLIPBOARD_SET, CLIPBOARD_GET, CLIPBOARD_CLEAR), WAIT, browser actions (NAVIGATE, NEW_TAB, CLOSE_TAB, SWITCH_TAB, REFRESH, BACK, FORWARD), COOKIE_MANAGEMENT, EXTENSION_INTERACTION, file actions (UPLOAD_FILE, DOWNLOAD_FILE, LIST_DIRECTORY, CHANGE_DIRECTORY, READ_FILE, WRITE_FILE, CONVERT_FILE), and BASH_COMMAND. ScrollDirection is up/down/left/right; MouseButton is left/right/middle.

5. REST API Surface#

The API Gateway (api_gateway/app.py) mounts routers under the configurable prefix /api/v1 (settings.api_prefix). Health routes are unprefixed. Responses use ORJSONResponse. GZip compression is applied to bodies over 1 KB. Middleware order (outermost first): RequestIDMiddleware, LoggingMiddleware, TimingMiddleware, CORS, GZip. No authentication middleware is currently installed — endpoints are open; the WebSocket token query parameter is accepted but not validated.

Health (unprefixed) — routers/health.py#

Health endpoints are unprefixed and available at the root of the service. The /ready endpoint aggregates dependency checks, though both the database and Redis checks are currently placeholders.

Features and Experiments#

These endpoints expose runtime feature flags and A/B experiment configuration. The /experiments/{name}/assign endpoint returns a deterministic variant assignment for a given identifier, using consistent hashing.

Personas — routers/personas.py#

The personas collection supports full CRUD plus lifecycle actions (activate, deactivate) and a clone operation. The 409 Conflict response prevents duplicate persona names within an organization.

Sessions — routers/sessions.py#

Session management endpoints follow a lifecycle: create → start → join-meeting → pause/resume → end. The /metrics endpoint returns aggregate counts across all sessions without pagination.

Knowledge — routers/knowledge.py#

The knowledge endpoints cover document lifecycle (create, upload, list, get, update, delete, reprocess) and search. The upload endpoint accepts multipart form data; the document type is inferred from the file extension.

Tools — routers/tools.py#

The tools endpoints manage registrations of callable integrations. The name and tool_type fields are immutable after creation. The /schema endpoint emits the tool's OpenAI-function-style JSON schema, suitable for passing to LLM APIs that accept function definitions.

Webhooks — routers/webhooks.py#

Webhook management includes delivery history, a test-fire endpoint, and secret rotation. The /rotate-secret endpoint returns the new secret once and never again — callers must store it immediately.

WebSocket management — routers/websocket.py#

These REST endpoints provide visibility into active WebSocket connections — useful for debugging and monitoring live session traffic.

The persona-service (persona-service/src/persona_service/main.py) is a separate FastAPI app exposing its own routes — see §9. It is unprefixed and not mounted by the gateway.

6. WebSocket Protocol#

api_gateway/routers/websocket.py exposes two WebSocket endpoints under the API prefix. The session-specific endpoint automatically subscribes the connection to all events for that session, making it the primary entry point for clients building real-time session integrations.

• GET /api/v1/ws — general connection. Accepts an optional token query parameter (not validated).
• GET /api/v1/ws/sessions/{session_id} — session connection; auto-subscribes to session_id on connect.

Message Envelope#

All messages are JSON objects with a type field (WSMessageType). Client→ server messages dispatched by MessageHandler.handle_message:

Server→client message types: event (SessionEvent), transcript (TranscriptEvent carrying a TranscriptSegment), metrics (MetricsEvent), error (ErrorMessage), ack (AckMessage), pong (PongMessage).

Control Commands#

MessageHandler registers handlers for all 15 non-subscription CommandType values: session start/pause/resume/end, audio mute/unmute, video enable/disable, screen share start/stop, AI interrupt/set-mode, tool execute/cancel, and persona switch. Session lifecycle commands broadcast a corresponding WSEventType to all session subscribers; tool.execute broadcasts tool.executing then tool.completed. Handlers log and return acknowledgement payloads — they do not yet invoke the orchestrator or tool-framework services.

TranscriptSegment fields#

TranscriptSegment is the payload type for transcript server→client messages. It carries timing metadata so clients can synchronize caption display with the avatar's speech output.

speaker_id, speaker_name, text, start_time, end_time (seconds, >= 0), confidence (0–1, default 1.0), is_final: bool, language (default en). emit_transcript() and emit_session_event() are exported helpers other services can call to broadcast over the connection manager.

7. Webhook Subsystem#

The delivery engine lives in api_gateway/services/webhooks.py. It handles signing, retry logic, and delivery records for all outbound webhook calls. The retry policy uses exponential backoff with jitter to avoid thundering-herd retries.

• Signing — HMAC-SHA256 over the JSON payload.
• Payload — WebhookPayload is { id, event_type, timestamp, api_version, data } with api_version defaulting to "2024-01-01".
• Retry — RetryConfig: max_attempts=5, initial_delay_seconds=1.0, max_delay_seconds=300.0, backoff_multiplier=2.0, jitter_factor=0.1, retryable status codes {408, 429, 500, 502, 503, 504}.
• Delivery status — DeliveryStatus: pending, delivering, success, failed, retrying, dead_letter.
• Records — DeliveryAttempt (per-attempt) and DeliveryRecord (all attempts) track outcomes.

The gateway's /webhooks/{id}/test endpoint currently simulates a successful 200 delivery rather than issuing a real HTTP request.

8. Session Orchestration Model#

The orchestrator service (services/psyche/orchestrator) owns the full session-lifecycle state machine and the real-time pipeline. Types are in orchestrator/session/types.py; logic in orchestrator/session/lifecycle.py. The orchestrator's session model is richer and more fine-grained than the API Gateway's — it tracks internal transition states (resuming, terminating) that the API does not expose to clients.

Session aggregate#

The Session is the central aggregate of the orchestrator. It carries all state needed to resume, pause, time out, or hand off a session.

SessionConfig carries session_type, priority, timeouts (idle_timeout_seconds=300, max_duration_seconds=3600, connection_timeout_seconds=30), feature toggles (enable_recording=False, enable_transcription=True, enable_screen_share=True, enable_tool_use=True), quality (video_quality/audio_quality default "high"), limits (max_participants=10, max_concurrent_tools=3), and persona linkage (persona_id, avatar_id, voice_id).

SessionState#

SessionState is a composite object capturing the evolving state of every dimension of the session. It is checkpointed periodically for recovery purposes.

Composed of ConversationState (phase, turn/message counts, topics, intents, short-term memory, key facts), EmotionalState (current_emotion, engagement_level, satisfaction_score, frustration_level, history), ToolState (active/pending/completed tool calls, computer-use and screen-share flags), and ScreenShareState. State carries a state_version and checkpoint metadata; StateCheckpoint snapshots SessionState with optional compression.

Lifecycle transitions (SessionLifecycleManager)#

The diagram below shows valid state transitions. Attempting an invalid transition (e.g. pausing a session that is not ACTIVE) raises a ValueError. The TimeoutManager polls every 10 seconds and fires warning and timeout callbacks.

                create_session()           start_session()
   (none) ──────────────────────▶ INITIALIZING ──────────────▶ ACTIVE
                                                                  │
                            pause_session()  ┌────────────────────┤
                                              ▼                    │
                                           PAUSED                  │
                                              │ resume_session()    │
                                              ▼                    │
                                         RESUMING ─────────────────┤
                                                                   │
                          terminate_session() / timeout            │
                                              ▼                    │
                          TERMINATING ──▶ TERMINATED ◀─────────────┘

pause requires status ACTIVE; resume requires PAUSED; start requires INITIALIZING — violations raise ValueError. TimeoutManager polls every 10s, firing a warning callback inside warning_before_timeout (60s) and a timeout callback (idle or max_duration). SessionHandoffManager models agent handoff with statuses pending/accepted/rejected/completed. The manager emits in-process events: session_created, session_started, session_terminated, session_paused, session_resumed, handoff_requested, handoff_completed.

Pipeline (orchestrator/pipeline/types.py)#

These enums describe the data flowing through the real-time processing pipeline and the services that consume it. They are used to label pipeline messages and route them to the correct destination service.

StreamType (audio_input/output, video_input/output, screen_share, data, control, events), PipelineStage (input, preprocessing, processing, postprocessing, output), MessageType (data, audio_frame, video_frame, text, command, event, response, error, heartbeat, ack), and ServiceType (asr, tts, llm, nlu, vision, avatar, emotion, knowledge, tools). The orchestrator also ships batching/, caching/ (semantic cache + embedding service), multiparty/ (participants, turns, sync), and resources/ (allocator, monitor, pool) submodules.

9. Persona Service Model#

services/psyche/persona-service is a standalone FastAPI app (persona_service/main.py, title "Psyche Persona Service") backed by the persona_system package. It is richer than the API Gateway's persona model and stores personas in an in-process PersonaSystem instance. The reason for this separation is that the Persona Service implements the full domain logic including OCEAN personality, expertise evaluation, memory storage, and constraint checking — capabilities that don't belong in the API Gateway's thin CRUD layer.

Endpoints (selected)#

• Templates: GET /templates/personas|personalities|domains|constraints.
• CRUD: POST/GET/PUT/DELETE /personas, GET /personas (filters organization_id, role, status, limit, offset), POST /personas/{id}/clone, POST /personas/from-template/{template_id}.
• Personality: GET/POST /personas/{id}/personality.
• Expertise: GET/POST /personas/{id}/expertise, POST /personas/{id}/expertise/domains, POST /personas/{id}/expertise/evaluate.
• Memory: POST /personas/{id}/memories/facts|opinions|interactions|retrieve.
• Constraints: GET/POST /personas/{id}/constraints, POST /personas/{id}/constraints/check-response, POST /personas/{id}/constraints/check-drift.
• Assets: POST /personas/{id}/avatars|voices, POST /avatars/{id}/train, POST /voices/{id}/train, POST /personas/{id}/assets/link, GET /personas/{id}/assets.
• Prompt: POST /personas/{id}/system-prompt.

Domain types#

These are the core domain types managed by the Persona Service. Unlike the API Gateway's flat models, these types carry the full behavioral configuration used by the LLM, voice, and avatar systems at session time.

• PersonaIdentity — id (UUID), name, display_name, role (PersonaRole), custom_role, gender_presentation, age_range, background_story, company_affiliation, job_title, status (PersonaStatus), created_at/updated_at, created_by, organization_id, tags.
• BigFivePersonality — the OCEAN model does exist here: openness, conscientiousness, extraversion, agreeableness, neuroticism, each a float defaulting to 0.5 with a validate() bounds check. CommunicationParameters adds style, formality_level, humor_level, expressiveness, response_length, verbosity, technical_level, and behavior flags. PersonalityConfig ties BigFivePersonality + CommunicationParameters + BehaviorMapping to a persona_id.
• DomainExpertise — domain_id, domain_name, level (ExpertiseLevel), coverage (0–1), confidence_threshold, topics, excluded_topics, knowledge_snippets, related_domains.
• Memory — PersonaFact (category, key, value, importance, is_public, use_count), opinion anchors, and per-user story events.
• Presets — PERSONALITY_PRESETS ships five presets (warm_professional, empathetic_helper, efficient_expert, enthusiastic_guide, calm_advisor). SYSTEM_TEMPLATES ships four persona templates (professional_assistant, friendly_support, sales_professional, technical_instructor). DOMAIN_TEMPLATES and CONSTRAINT_PRESETS are also exported.

10. Persistence#

Database bootstrap#

services/psyche/api-gateway/src/api_gateway/db/migrate.py is the only database-DDL code in the domain. It is an idempotent bootstrap script (not Alembic migrations) intended to run as a Kubernetes Job. Its scope is narrow: it prepares the PostgreSQL extensions and creates the audit schema, but does not create any domain tables. Specifically, it:

• Enables extensions: uuid-ossp, pgcrypto, vector (pgvector), pg_trgm, btree_gin, btree_gist.
• Creates schemas: persona, knowledge, sessions, analytics, audit.
• Creates one table — audit.change_log (id, table_name, operation, old_data JSONB, new_data JSONB, changed_by, changed_at, session_id, ip_address INET, user_agent) — with four indexes.
• Defines two PL/pgSQL functions: update_updated_at_column() and audit_changes().

No personas, sessions, knowledge_documents, tools, or webhooks tables are created. The module notes that Alembic migrations are not yet introduced.

Runtime storage#

The API Gateway routers store every resource in module-level Python dicts (_personas, _sessions, _documents, _tools, _webhooks, _deliveries) with integer counters generating IDs. Each router file states "In-memory storage for demo purposes." app.py's lifespan handler leaves database and Redis initialization commented out. The persona-service likewise keeps state inside an in-process PersonaSystem.

Configured stores#

The API Gateway Settings declares a PostgreSQL URL (asyncpg driver) and a Redis URL with pool sizes, and pyproject.toml includes sqlalchemy[asyncio] and asyncpg. These dependencies are present but the gateway does not yet open connections at runtime.

11. Avatar Engine Configuration#

services/psyche/avatar-engine/src/avatar_engine/config.py defines a nested pydantic-settings configuration. The package implements 3DGS and NeRF rendering (rendering/, nerf/), FLAME fitting (preprocessing/, models/flame.py), training (training/), quantization, lip sync (lipsync/), expression/eye/body systems, and cultural overlays.

RenderConfig (env prefix AVATAR_RENDER_)#

RenderConfig controls real-time rendering parameters. Note that the configuration defaults to internal gRPC ports (50051/50052), while the Dockerfile overrides the HTTP service port to 8001 — these are different protocol endpoints on the same service.

Additional groups cover ambient occlusion, dynamic resolution, and multi-resolution rendering.

TrainingConfig (env prefix AVATAR_TRAIN_)#

TrainingConfig governs the offline 3DGS avatar training pipeline. The densification settings control when the system adds or prunes Gaussian splats to match the training video.

iterations=30000 (1000–100000), batch_size=1 (1–8), learning_rate=1e-4, per-attribute Gaussian learning rates, densification settings (densify_from_iter=500, densify_until_iter=15000, densify_grad_threshold=0.0002), loss weights (lambda_l1=0.8, lambda_ssim=0.2, lambda_lpips=0.0, lambda_reg=0.01), checkpointing, and early stopping.

FLAMEConfig (env prefix AVATAR_FLAME_)#

FLAMEConfig controls the FLAME parametric face model that provides the expression-control skeleton underlying the 3DGS render.

model_path/landmark_path, n_shape=100 (10–300), n_expr=50 (10–100), n_tex=50 (10–100), and shape/expression/pose regularization weights.

PreprocessingConfig (env prefix AVATAR_PREPROCESS_)#

PreprocessingConfig controls the face detection and video preprocessing step that prepares training data for 3DGS optimization.

face_detector="mediapipe" (also mtcnn, retinaface), num_landmarks=68, min_video_duration=5.0, max_video_duration=300.0 (cap 600.0), background_model="robust_video_matting", quality thresholds.

AvatarConfig (root, env prefix AVATAR_)#

The root avatar service configuration, which composes the four configs above.

service_name="avatar-engine", service_port=50051, grpc_port=50052, device=cuda (DeviceType), gpu_id=0, enable_mixed_precision=true, and the four nested sub-configs above. Note: the avatar-engine config defaults to ports 50051/50052, while its Dockerfile sets SERVICE_PORT=8001 (see §16).

12. Voice Engine Model#

services/psyche/voice-engine implements TTS, STT, diarization, phoneme/viseme generation, turn-taking, captions, transcript processing, and audio handling. The service is organized into subpackages, each responsible for a distinct stage of the voice pipeline.

TTS providers (tts/)#

Each TTS provider is a separate module implementing the TTSProvider abstract interface. A provider_chain.py implements ordered failover across providers. Supporting modules handle advanced synthesis features.

Provider modules: elevenlabs.py, openai_tts.py, cartesia.py, deepgram_aura.py, orpheus.py, f5_tts.py, piper.py. A provider_chain.py implements failover. Supporting modules: voice_cloning.py, style_transfer.py, emotional_synthesis.py, emotion_intensity.py, personality_variation.py, voice_quality.py, voice_similarity.py, watermarking.py, phrase_cache.py, usage_audit.py, streaming.py.

SynthesisConfig (tts/types.py) is the synthesis request: voice_id / voice_profile, output_format (AudioFormat, default pcm_16), sample_rate (default 24000), channels, speaking_rate (0.5–2.0), pitch_shift (semitones), volume_gain_db, emotion (VoiceEmotion), style_intensity, optional dynamic-emotion-intensity, optional VoicePersonalityTraits (formality, friendliness, verbosity, humor, empathy, assertiveness — modeled after the API persona traits), SSML fields, voice-cloning reference audio, style-transfer reference audio, watermark_payload, audit_metadata, streaming (chunk_size_ms=100), timeout_seconds=30.0, and quality (standard/high).

AudioChunk carries data: bytes, format, sample_rate, channels, duration_ms, sequence_num, is_last, with numpy conversion helpers. SynthesisResult adds timing (processing_time_ms, first_byte_latency_ms), provider, voice_id, text_length, cost_usd. TTSProvider is the abstract provider interface (synthesize_stream, synthesize_streaming_text, synthesize_batch, health_check, get_voices, close).

STT providers (stt/)#

The STT subsystem mirrors the TTS structure: individual provider modules, a provider_chain.py for failover, and supporting modules for accuracy improvements.

Provider modules: deepgram.py, whisper.py, local_whisper.py, assemblyai.py, google_speech.py, plus provider_chain.py, punctuation_restoration.py, and fine_tuning.py. ProviderType (STT) is DEEPGRAM, WHISPER, ASSEMBLYAI.

Other subsystems#

The voice engine is a full-featured pipeline with dedicated subpackages for each processing stage. Each subpackage covers a distinct concern.

conversation/ (LLM, RAG, context, prompts, long-term memory, style), diarization/, phoneme/ (g2p, IPA, alignment), viseme/, turn_taking/ (endpointing, interruption, timing), captions/, transcript/ (confidence, correction, entities, intent, sentiment, streaming, timestamps), sync/ (AV sync), failover/ (circuit breaker, health), commands/, and audio/ (VAD, echo cancellation, noise suppression, gain control, resampling, spatial audio).

13. Perception Engine Model#

services/psyche/perception-engine implements emotion, intent, and screen perception (emotion/, intent/, screen/, context/). The emotion types are the most detailed — the perception engine uses a richer set than the simple emotion states in the orchestrator, because it models the full affective state of participants including valence (positive/negative) and arousal (high/low).

Emotion types (emotion/types.py)#

• EmotionType — 7 emotions plus neutral (see §4). EmotionProbabilities holds a dict[EmotionType, float] distribution with dominant_emotion, confidence, and entropy properties.
• BoundingBox — x, y, width, height, confidence, with center, area, IoU helpers.
• FacialLandmarks — a 68-point landmark array with named regions and derived metrics (eye-aspect-ratio, mouth-aspect-ratio, eyebrow-height-ratio).
• VideoEmotionResult, AudioEmotionResult, FusedEmotionResult — modality results; AudioFeatures carries prosodic/energy/spectral/voice-quality features (pitch, energy, MFCCs, jitter, shimmer, HNR).
• EngagementMetrics — engagement_score, attention_score, confusion_score, interest_score, looking_at_camera, gaze_deviation, head_nods, head_shakes, blink_rate.
• EmotionDetectorConfig — face_detector_backend="retinaface", landmark_model="dlib_68", emotion_model="fer2013", temporal smoothing, and modality-fusion weights (video_weight=0.6, audio_weight=0.4).

14. Computer-Use Action Model#

services/psyche/computer-use implements an agentic browser/desktop automation loop. The service is organized into subpackages covering the Observe-Reason-Act loop, vision capabilities, and safety guardrails.

Subpackages: actions/, agent/ (loop, planning, recovery, undo, dead-end, explain), vision/ (OCR, screenshot, UI detection, semantics, accessibility), environment/ (container, display, Windows VM), screen_sharing/, demonstration/, and benchmark/.

Action (actions/types.py) is the core type: every browser or desktop operation is represented as an Action instance. Factory constructors exist for each common action type, avoiding the need to manually set all optional fields.

Action (actions/types.py) carries an action_type (ActionType, see §4) plus optional mouse coordinates (Coordinate/end_coordinate), keyboard parameters (text, key), clipboard_selection, scroll parameters (scroll_direction, scroll_amount=3), duration_ms, browser parameters (url, tab_index), file_path, and command. Factory constructors exist for each common action. ActionResult records success, action_type, screenshot_base64, output, error, duration_ms, cursor_position, and converts to Claude tool-result format. ActionHistory keeps a bounded history (max_size=100). SPECIAL_KEYS and MODIFIER_KEYS map names to X11 keysyms.

15. Knowledge Base Model#

services/psyche/knowledge-base implements ingestion, embedding, RAG, and retrieval — distinct from the gateway's in-memory document store. This service is the backend that the gateway search endpoint will eventually call; currently they operate independently.

• ingestion/ — DocumentType (pdf, docx, doc, html, text, markdown, excel, csv, powerpoint, json, audio, video, image, unknown), ProcessingStatus (pending, processing, chunking, embedding, indexing, completed, failed), ChunkingStrategy (fixed_size, semantic, sentence, paragraph, hierarchical, document_specific), plus parsers, cleaning, and chunking modules.
• embedding/ — embedding generators, vector stores, and a manager.
• rag/ — pipeline, generator, evaluator, prompts.
• retrieval/ — query, search, reranking, context, pipeline.
• management/ — analytics, lifecycle, quality, updates.

The knowledge-base service's ProcessingStatus (7 stages) differs from the API Gateway's DocumentStatus (4 states) and from DocumentType in models/knowledge.py (7 types). They are separate type sets.

16. Service Ports#

Ports below are taken from each service's Dockerfile (SERVICE_PORT). These are the ports services listen on within the Kubernetes cluster. External traffic enters exclusively through the API Gateway on port 8000.

The avatar-engine application config (AvatarConfig) defaults to 50051/50052 for direct/gRPC use; the Dockerfile overrides the HTTP service port to 8001. Services psyche-tavus-integration, psyche-video-conferencing, psyche-security, and psyche-error-handling do not declare a Dockerfile port.

17. API Gateway Configuration#

Settings (api_gateway/config.py) uses pydantic-settings, reading a .env file (case-insensitive, extra keys ignored). The table below lists key fields and their defaults. Note that the configured internal URLs for orchestrator (:8003) and knowledge_base (:8004) do not match those services' Dockerfile ports (:8007 and :8008) — this discrepancy must be resolved when wiring the gateway to the backend services.

The configured internal-service URLs for orchestrator (:8003) and knowledge_base (:8004) do not match those services' Dockerfile ports (:8007 and :8008).

Feature flags#

Feature flags control which optional capabilities are enabled at runtime. They are exposed as environment variables and served to clients via GET /api/v1/features.

feature_computer_use (default true), feature_voice_cloning (true), feature_multi_party (false), feature_screen_share (true). Exposed via GET /api/v1/features as FEATURE_COMPUTER_USE, FEATURE_VOICE_CLONING, FEATURE_MULTI_PARTY, FEATURE_SCREEN_SHARE.

Experiments#

The experiments system supports A/B testing via deterministic variant assignment. Each experiment defines a set of named variants with integer weights that must sum to 100.

experiments is a dict[str, dict[str, int]] of variant→weight maps; a validator enforces that each experiment's weights are non-negative integers summing to 100. experiment_salt (default serwaa) feeds deterministic bucketing (assign_variant in api_gateway/experiments.py), producing a bucket 0–99.

The avatar engine, perception engine, and other services each define their own pydantic-settings configuration with their own env prefixes (e.g. AVATAR_*). There is no single shared environment-variable contract across services.

18. Shared Contracts Package#

libs/contracts/psyche (@psyche/contracts) is a Zod-based TypeScript contracts package intended to be consumed by any TypeScript client or library that needs type-safe access to Psyche data types. Only the common module is currently implemented; per-domain contract modules (avatar, voice, behavior, etc.) are declared in package.json exports but their source files do not yet exist.

libs/contracts/psyche (@psyche/contracts) is a Zod-based TypeScript contracts package. Its package.json declares exports for ./avatar, ./voice, ./behavior, ./perception, ./conferencing, ./knowledge, and ./persona, but only src/common/index.ts and src/index.ts exist — src/index.ts re-exports common and keeps the per-domain exports commented out pending migration.

The implemented common module exports these Zod schemas and inferred types:

The contracts package's SessionStatusSchema (initializing/active/paused/ ended/error) is yet another distinct session-status value set, matching neither the API Gateway's nor the orchestrator's enum.

19. V2 Fighting-Game Cross-Domain Contracts#

The V2 fighting-game project (V2/) consumes three distinct Psyche capability clusters through dedicated @v2/* adapter packages. Each adapter wraps a Psyche library, runs entirely on V2's client/services tier, and is bound by the same hard safety property: none of these outputs may ever feed the deterministic, rollback-networked combat simulation. They are presentation, accessibility, and compliance surfaces only. The TypeScript contracts encode this as a literal mayInfluenceRollback: false field, and V2's V2Netcode modules are statically forbidden from referencing any of these packages.

These sections describe what is implemented in source under apps/v2/psyche-ai-director-hints/, apps/v2/psyche-caption-streaming/, apps/v2/eu-ai-act-surface/, and the underlying Psyche libraries libs/psyche/behavior-prediction, libs/psyche/caption-streaming, and libs/psyche/action-safety.

19.1 V2 Adaptive AI Director Hints Contract#

@v2/psyche-ai-director-hints is the V2 adapter between the V2AdaptiveAI Unreal plugin and @psyche/behavior-prediction. It is implemented in apps/v2/psyche-ai-director-hints/src/psyche-ai-director-hints.ts and depends on @psyche/behavior-prediction (workspace:*), instantiating the predictor through createBehaviorPredictionEngine. The package exports the constants V2_ADAPTIVE_AI_PLUGIN_NAME ('V2AdaptiveAI') and V2_PSYCHE_BEHAVIOR_PREDICTION_PACKAGE ('@psyche/behavior-prediction'), and the plan builders buildV2AdaptiveAILiveHintPlan, buildV2AdaptiveAIMatchStartSnapshotPlan, and queueV2AdaptiveAIMidMatchPsycheUpdateForNextMatch.

The behavior-prediction engine produces per-player tendency hints — a five-axis V2AdaptiveAITendencyVector (pressure, defense, spacing, throwGame, resourceUse) plus a next-action prediction and confidence. How those hints reach the AI Director depends on the match mode, captured by the V2AdaptiveAIHintConsumptionMode union:

1. non-rollback-live. In modes that do not use rollback networking (training, single-player, lobby practice), V2AdaptiveAI consumes live Psyche hints each tick. The hint is advisory CPU-behavior shaping; it is not a simulation input shared between peers, so live RPCs are permitted.
2. rollback-match-start-snapshot. In rollback-with-CPU modes, the adapter samples Psyche behavior-prediction once at match load and bakes the result — the tendency-vector + seed — into the deterministic match-start input vector. After that snapshot, the CPU runs from the seeded vector with no further Psyche RPCs, so every peer re-simulates identically during a rollback. The source marks this path rollback-match-start-snapshot and tags the serialized payload v2NetcodeTreatsAsOrdinaryDeterministicInput.

The contract enforces the rollback boundary with two complementary guards: non-rollback-live plans are rejected when the match context reports calledFromRollbackFrame, and the spec asserts that the adapter rejects live Psyche RPCs from rollback frames ("Psyche RPC is rejected inside rollback frames"). Mid-match Psyche updates that arrive during a rollback match are not applied to the running sim; they are deferred by queueV2AdaptiveAIMidMatchPsycheUpdateForNextMatch and folded into the next match-start snapshot instead. The rollback policy string is live-off-rollback-or-match-start-snapshot-only, and every hint and plan carries mayInfluenceRollback: false.

This contract is registered in V2's services layer as the psyche-behavior-prediction adapter under EV2OshunDomain::Psyche, mirrored in the V2AdaptiveAI plugin types (FV2AdaptiveAITendencyHint, FV2AdaptiveAIMatchStartInputVector), and validated by V2/ue/Tools/check-v2-adaptive-ai-psyche.py together with the repository's rollback-determinism self-test.

19.2 V2 Caption Streaming Contract#

@v2/psyche-caption-streaming wires real-time V2 match captions through @psyche/caption-streaming. It is implemented in apps/v2/psyche-caption-streaming/src/psyche-caption-streaming.ts, depends on @psyche/caption-streaming (workspace:*), and composes the Psyche accessibility caption engine and live orchestrator via createAccessibilityCaptionEngine and createLiveMeetingOrchestrator. The exported constants name both ends of the bridge: V2_PSYCHE_CAPTION_STREAMING_SOURCE_PACKAGE ('@psyche/caption-streaming'), V2_PSYCHE_CAPTION_STREAMING_PACKAGE_NAME ('@v2/psyche-caption-streaming'), its binding id V2_PSYCHE_CAPTION_STREAMING_BINDING_ID, and the accessibility owner V2_IRIS_ACCESSIBILITY_BRIDGE_PACKAGE ('@v2/iris-accessibility').

buildV2PsycheCaptionStreamingSurface builds a per-match plan across five caption surfaces (in-match-hud, spectator-overlay, replay-viewer, companion-second-screen, broadcast-observer). The surface flags advertise its scope: controlsRealTimeMatchCaptions: true for in-match HUD captions, controlsAccessibilityCaptions for the accessibility caption channel, and controlsSpectatorModeCaptions for the spectator mode overlay that broadcasts and tournament observers consume. Replay support is exposed through supportsReplayCaptionTranscript, and live channel changes are published on the v2.caption-stream.spectator.channel.updated event topic.

Captions are strictly a UI presentation layer. The rollback policy is off-rollback-caption-ui-only, the surface sets mayInfluenceRollback: false, and the bridge never writes deterministic match-simulation state. Iris remains the accessibility-semantics owner (the caption channel is reused by @v2/iris-accessibility as its real-time caption provider); Psyche provides the caption-engine implementation until Iris owns live captions end to end. The contract is validated by V2/ue/Tools/check-v2-psyche-caption-streaming.py.

19.3 EU AI Act Transparency Surface (@psyche/action-safety)#

Psyche owns the player-facing classifier transparency and opt-out half of V2's EU AI Act compliance surface. The implementation lives in libs/psyche/action-safety/src/v2-ai-act-transparency.ts (package @psyche/action-safety) and exports buildV2AdaptiveAIClassifierTransparency and evaluateV2AdaptiveAIOptOut. The transparency record classifies the Adaptive AI Director as a limited-risk AI system, carries the human-readable disclosure text, names the classifier purpose and signals, and links the hosted Model Card slug adaptive-ai-director. Opt-out is always available (optOutAvailable: true): when a player opts out, evaluateV2AdaptiveAIOptOut returns a decision that disables adaptive behavior and falls back to a static-cpu-profile (fallbackMode: 'static-cpu-profile'). Like every other V2 surface, the transparency record and the opt-out decision are off-rollback (offRollback: true, mayInfluenceRollback: false).

This Psyche transparency contract is one of three composed by the @v2/eu-ai-act-surface service package. That package depends on @psyche/action-safety for transparency/opt-out, on @nous/safety for Model Card hosting (buildV2AdaptiveAIModelCard and the AI-commentary / anti-cheat / generation-pipeline Model Cards), and on @themis/accountability for the AI-system-of-record registration and the regulator-ready conformity export. The composed surface marks AI commentary and generation outputs as off-rollback presentation content and applies the player opt-out before any adaptive classifier runs. The cross-domain surface is validated by V2/ue/Tools/check-v2-eu-ai-act-surface.py; the Nous and Themis halves of the contract are documented in DOMAINS/nous/specifications.md and DOMAINS/themis/specifications.md.

Acceptance Criteria#

A change to a Psyche Python service is acceptable when:

1. Type safety — every API model derives from BaseSchema (gateway) or is a typed dataclass / pydantic model; extra="forbid" is preserved on gateway models.
2. Enum fidelity — new states or types extend the relevant enum in §4 rather than introducing free-form strings.
3. Lint and types — ruff and mypy pass for the touched service.
4. Tests — pytest passes; each service has a tests/ directory.
5. Build — TypeScript libraries build with tsup and pass vitest.
6. No fabricated capability — documentation and OpenAPI metadata describe only behavior that the code performs (e.g. the gateway's knowledge search is title-matching, not vector search).