Psyche — Real-Time Runtime Substrate

Psyche is the V1 real-time embodiment substrate: it owns the contract for a live text / voice / avatar session — the session envelope, server-mediated turn-taking, interruption and barge-in, transcript synchronization, lip-sync and expression coherence, latency budgeting, provider failover, and continuity under reconnect or a safety crisis. It serves every surface where a member talks with an assistant in real time rather than reading a response (the live assistant shell, the embodied teacher personas, conferencing, and the Living Scenes frame stream). It sits among the platform-substrate deep-dives hubbed at ../ARCHITECTURE.md, alongside Sophia, Iris, Lilith, Isis, and Aje.

A deliberate split runs through this page, and it is the single most important thing to understand about Psyche's maturity. The contract-and-logic layer — the pure data shapes and pure functions that define what a correct session looks like — is real, rich, and exhaustively tested in libs/oshun/embodiment-psyche. The live wire/transport binding and the actual provider integration (real ASR/TTS/avatar vendors, the WebSocket data plane, the operator review lane) are spec-only / partially built: the adapter library carries no IO, no rendering, and no transport by design. The end-to-end "tutor live session to graded record" walkthrough is rated partial in the completeness audit, with the live-voice envelope, library write, and operator review lane noted as unbuilt surfaces. Where this page describes a function or constant, it exists; where it describes a deployed live session, it is honestly labeled planned or gated.

Canonical home (§13). Psyche is a cross-product substrate, so its canonical reference home is the domain space docs/domains/psyche and its code-linked entity catalog at systems/psyche. This page is V1's view — how the V1 platform composes Psyche; the substrate itself is documented in full at its canonical home, which this page references rather than duplicates.

Where Psyche lives in the codebase#

@oshun/embodiment-psyche (libs/oshun/embodiment-psyche, v0.1.0) is the canonical contract library. It is type: module, and its main/types point at ./src/index.ts — source, not a built dist — so consumers import the contract directly. src/index.ts re-exports 23 modules, which is the real scope of the substrate; the older architecture note that named only adapter.ts, avatar-sync.ts, backpressure.ts, and crisis-frame.ts materially undersells it.

The service backing — the live runtime that consumes these contracts — exists with substantive content but is the less-mature half: services/psyche/* (real subdirectories include orchestrator, avatar-engine, behavior-engine, voice-engine, conferencing/video-conferencing, tavus-integration, perception-engine, tool-framework, persona-service, …), infrastructure/psyche/{docker,kubernetes,terraform}, contracts under libs/contracts/psyche/src, and apps/psyche/admin (only the admin app is present). Separately, libs/psyche is a much larger substrate tree (~139 subdirectories, including tavus-*, avatar-*, memory-*, viseme-generator, voice-synthesis, speech-recognition) — this is not the same thing as the embodiment-psyche adapter contract; do not conflate them.

The canonical adapter — pure logic over an injected port#

createCanonicalPsycheEmbodimentAdapter({ apiAdapter }) is the canonical entry point. It takes a PsycheEmbodimentApiAdapter — the low-level port that does the IO — and returns a PsycheEmbodimentAdapter that layers the canonical contract logic on top: getContractDescriptor, getMetadata, getAvailability (derived from the injected getHealth), getSessionCapabilities, resolveEmbodimentProfile, getLiveSessionState, planSession, startEmbodiedSession, and the pause/resume/terminate lifecycle calls.

The split is the whole point. adapter.ts contains zero fetch, WebSocket, http, or prisma calls — verified by grep. startEmbodiedSession illustrates the discipline: it resolves the embodiment profile, builds a session plan via buildPsycheSessionPlan, and refuses to proceed if the plan is not approved (if (!plan.approved) throw …). Only then does it hand the granted modalities and session kind to the injected apiAdapter.createSession. Policy and shape are decided in pure code; the network call is delegated. This is why the library is real and rich while a deployed live session is still partial: the rules are implemented; a running session requires the transport and provider bindings that live downstream in services/psyche/*.

The session envelope#

The session envelope (session-envelope.ts, PsycheSessionEnvelope at line 393) is a single validated record that both endpoints of a live session agree on. It is far richer than a connection descriptor — it carries identity, residency, capability grants, entitlement, policy binding, transport, latency budget, continuity, governance, and lifecycle in one fingerprinted object.

Key fields:

• sessionId, version, mode (PsycheSessionMode, an alias of PsycheSessionKind).
• assistantIdentityId + assistantIdentityFingerprint — the Iris assistant identity driving the session, with a fingerprint so both ends can confirm they are bound to the same identity.
• locale, region (residency/routing), timezone, deviceClass (web/mobile/wearable/desktop/unknown), platformShell, consumer.
• capabilities — the wire/runtime capabilities the server may emit (PSYCHE_SESSION_CAPABILITIES, 21 entries, including lip-sync, emotional-modulation, screen-context, and translation).
• entitlementClass — PSYCHE_SESSION_ENTITLEMENTS = free/premium/enterprise/operator-admin/operator-studio.
• lilithPolicyVersion — the Lilith policy pack bound at session open.
• embodiment, transport, latencyBudget, governance, and continuity.
• continuity.memoryScope — PSYCHE_SESSION_MEMORY_SCOPES = off/session/profile — plus memoryConsentGranted, groundingMode (none/recommended/required), activeDomain, conversationHistoryId, and carryOverContextId, which is how an Iris conversation carries into a live session.
• lifecycleState, timestamps, reconnectAttempts, lastError, and a deterministic fingerprint.

The envelope is not a passive bag of fields — the module ships the operations that keep it correct:

• Lifecycle as a transition graph. PSYCHE_SESSION_LIFECYCLE_TRANSITIONS encodes the legal moves between states (pending → ready → connecting → connected, with degraded reachable from connecting/connected and recoverable back to connected, and the terminal set closed/failed/timed-out). isPsycheSessionTransitionAllowed and isPsycheSessionTerminalState enforce it. A session cannot skip from pending straight to connected.
• Fingerprinting. computePsycheSessionEnvelopeFingerprint derives a deterministic hash over the identity-relevant fields, so two endpoints holding matching fingerprints know they are looking at the same envelope without reshipping the whole record.
• Modality negotiation. negotiatePsycheSessionModalities reconciles requested modalities against grants and re-fingerprints the result.
• Persona-switch planning. planPsycheSessionPersonaSwitch plans a mid-session persona change while preserving continuity.

The event taxonomy and the gap-free stream#

Psyche ships a dual event taxonomy, and the architecture note that only listed the 15 canonical V1 wire types under-describes it.

Canonical V1 wire events — PSYCHE_V1_SESSION_EVENT_TYPES (session-events.ts:115), 15 entries that match the published list exactly: text-token-stream, asr-partial, asr-final, tts-chunk, avatar-frame, viseme-stream, expression-update, turn-complete, interruption, reconnect, transcript-sync, error, kill-switch, fallback-engaged, policy-intervention.

Server turn events — PSYCHE_SERVER_TURN_EVENT_KINDS = turn-start, turn-end. These are emitted by the server-side turn reducer (below) and are not in the 15-entry V1 list, which is a documentation gap rather than a contradiction.

Legacy dotted protocol — PSYCHE_LEGACY_SESSION_EVENT_KINDS is the older namespaced family that the union still carries:

• Turn-taking: turn.claim, turn.grant, turn.yield, turn.interrupt, turn.barge-in, turn.timeout
• Streaming: stream.audio.frame, stream.transcript.partial, stream.transcript.final, stream.avatar.frame, stream.tool.call, stream.tool.result
• Reconnect: reconnect.attempt, reconnect.success, reconnect.failed, reconnect.handoff
• Lifecycle: session.state-changed, session.error, session.heartbeat

The discriminated union PsycheSessionEvent (and the combined PSYCHE_SESSION_EVENT_KINDS) covers all three families.

Every event flows through a gap-free, monotonic stream. buildPsycheSessionEventStream / appendPsycheSessionEvent enforce that sequence === nextSequence (no gaps, no reordering) and that emittedAt is non-decreasing (monotonic time). Each event is fully traceable, carrying an eventId, sessionId, traceId, timing (ingress/egress plus provider attribution and retryAttempts), the sequence, and an actor. This is what makes the stream auditable and replayable: a kill-switch or policy-intervention event has a stable position and a provable provenance.

Server-mediated turn-taking — a real state machine#

Turn-taking is not narrative hand-waving; it is an implemented reducer. createPsycheServerTurnState builds the initial state and applyPsycheServerTurnCommand advances it. The phases are PsycheServerTurnPhase = idle | active | interrupted | complete, and the commands are turn-start, turn-end, interruption, and system-initiated-barge-in:

The server is the authority on whose turn it is — barge-in and interruption are mediated through this reducer rather than negotiated peer-to-peer, which is what keeps a multi-party live session deterministic. The interruption commands fade out any in-flight TTS over PSYCHE_DEFAULT_PARTIAL_TTS_FADE_OUT_MS = 180 (session-events.ts:768) by default — the configurable barge-in fade-out that lets the synthetic voice tail off cleanly instead of cutting dead mid-word.

Latency budgets and the "thinking indicator" deadline#

Live embodiment lives or dies on latency, so the targets are codified, not aspirational prose (latency-dashboard.ts:68-84):

Sample builders (e.g. buildPsycheVoiceFirstAudioLatencySample, buildPsycheAvatarFrameLatencySample) compute each value from raw timestamps and validate ordering (first-audio cannot precede speech-end), and assessPsycheLatencyAgainstBudget compares observed percentiles to the envelope's budget, returning PsycheLatencyBudgetStatus (PSYCHE_LATENCY_BUDGET_STATUSES = ok / warning / exceeded) for the round trip, the pipeline, and overall. A warning/exceeded status is the input that feeds quality-driven fallback.

When a provider lags, the runtime must not leave the member staring at silence. The provider-backpressure policy PSYCHE_PROVIDER_BACKPRESSURE_DEFAULT_POLICY sets thinkingIndicatorDeadlineMs: 200 (backpressure.ts:322), clamped to ≤ 200 ms — so a "thinking" indicator must surface within 200 ms of a stall, which matches the published "within 200 ms" requirement.

Backpressure: admission control and load shedding#

Beyond the thinking-indicator deadline, backpressure.ts implements real load-management mechanics:

• Admission control. decidePsycheSessionAdmission returns a typed decision (PSYCHE_SESSION_ADMISSION_DECISIONS) weighing PSYCHE_SESSION_ADMISSION_PRIORITIES — i.e. whether a new live session may start under current load.
• Shed planning. planPsycheBackpressureShed picks running sessions to relieve, choosing among PSYCHE_BACKPRESSURE_SHED_ACTIONS = downgrade-to-text, graceful-disconnect, queue-handoff.
• Provider backpressure response. planPsycheProviderBackpressureResponse produces a throttle / thinking-indicator / fallback plan from the default policy (the ASR-throttle ratio, model-streaming-throttle ratio, and fallbackToTextAfterMs all live in that policy).
• Status rollup. computePsycheBackpressureStatus reduces the picture to PSYCHE_BACKPRESSURE_STATUSES = healthy / elevated / critical.

The fallback cascade#

When quality, latency, a provider failure, a kill-switch, a policy, or an accessibility need degrades a modality, Psyche steps down a fixed cascade rather than dropping the session. PSYCHE_FALLBACK_MODES (fallback-routing.ts:55) = avatar → voice → text → unavailable, and the triggers are PSYCHE_FALLBACK_TRIGGERS = provider-failure, quality, latency, kill-switch, policy, accessibility.

resolvePsycheFallbackMode ranks the requested mode against current capability and resolves the highest mode that can actually be served. buildPsycheFallbackCascadePlan and planPsycheFallbackRenegotiation produce the step-down plan and the user-facing decline message, and a fallback-engaged event is emitted onto the stream so the degradation is visible and audited. The kill switch has its own granularity: PSYCHE_KILL_SWITCH_SCOPES = session / tenant / region / global, with PSYCHE_KILL_SWITCH_FALLBACK_MODES = voice / text / closed — so an operator can collapse all avatar synthesis in a single region to text-only without taking text away.

Avatar lip-sync and expression coherence#

avatar-sync.ts makes "viseme alignment" concrete. PSYCHE_AVATAR_VISEMES is an 11-viseme set (rest, ah, ee, oh, oo, mb, fv, ss, th, eh, ay), and PSYCHE_AVATAR_VISEME_TABLE maps 39 ARPABET phonemes onto those visemes (resolvePsycheAvatarViseme upper-cases the phoneme id and looks it up). The pipeline is:

• buildPsycheAvatarPhonemeTimeline orders phonemes into a timed sequence.
• interpolatePsycheAvatarPose linearly interpolates between viseme poses so the mouth transitions smoothly rather than snapping.
• assessPsycheAvatarAlignment measures the gap between the avatar's actual pose and the audio it should track, returning PSYCHE_AVATAR_ALIGNMENT_STATUSES = aligned / drifting / desynced — drift detection so the runtime knows when lip-sync has come unstuck.

Expression coherence is handled alongside in emotion-modulation.ts, and screen-context.ts carries screen-share context for sessions that share a screen.

Crisis-frame continuity — a first-class safety boundary#

crisis-frame.ts is a fully implemented module, not narrative. A crisis-frame entry is a safety boundary, not a normal persona transition (its own fileoverview says so): when Lilith policy takes over for a member in distress, the runtime must break persona, stop synthetic output, and suspend memory writes at the same timestamp.

enterPsycheCrisisFrame produces a new envelope and the audit/event trail in one deterministic step:

• Sets continuity.memoryScope → 'off', embodiment.syntheticVoice → false and syntheticAvatar → false, clears the voice/avatar pack ids, strips the synthesis capabilities, and moves lifecycleState → 'degraded'.
• Emits a policy-intervention event onto the stream by actor { kind: 'system', id: 'lilith' }.
• Writes a persona.break audit event and a lilith.intervention audit event, both stamping the triple action PSYCHE_CRISIS_FRAME_AUDIT_ACTIONS = ['break-persona', 'halt-synthesis', 'suspend-memory-writes'].
• Produces a synthesis-halt plan (the tts/avatar pipeline services and voice/avatar/video modalities are removed) and re-validates the new envelope before returning.

The user-visible disclosure is fixed and honest: "I need to pause the persona voice and switch to direct safety support for this moment." The point is that a crisis cannot half-happen — persona break, synthesis halt, and the memory-write suspension are computed together, so Iris never records anything said inside a crisis frame.

Living Scenes integration#

The events/ subtree wires Psyche into Living Scenes: events/scene-events.ts carries scene events, events/cue-plan-replay.ts handles cue-plan replay, and events/frame-stream.ts is a real backpressure channel for the render frame stream. frame-stream.ts ships a DEFAULT_FRAME_STREAM_CONFIG, an evaluateBackpressure step, and stepFrameStream/consumeFrame, with a high-water policy that drops non-keyframes first when the channel saturates — which is how render envelopes are kept inside Living-Scenes latency budgets without tearing the scene.

Maturity, honestly#

To restate the split so nothing is oversold:

• Real and rich (contract + logic): the 23-module adapter library — session envelope and its lifecycle graph, the full event taxonomy and gap-free stream, the server turn-state reducer, latency budgets and assessment, admission / shed / provider backpressure, the fallback cascade and kill switch, the viseme table and alignment drift detection, and crisis-frame continuity. Every claim on this page maps to an exported symbol or constant, and the modules ship with comprehensive *.test.ts suites.
• Spec-only / partially built (runtime): the live WebSocket transport binding and the real ASR/TTS/avatar provider integrations (deliberately absent from the pure-logic library), and the end-to-end "tutor live session to graded record" flow — whose live-voice envelope, library write, and operator review lane are noted as unbuilt surfaces. The service trees under services/psyche/*, infrastructure/psyche/*, and apps/psyche/admin exist with substantive content but are the less-mature half. See the backlog (§11) for the live-session and operator-lane work that closes this gap.

Related#

• Lilith — Contemplative Policy Substrate
• Iris — Assistant Memory Substrate
• Living Scenes
• Sophia — Grounding Substrate
• Persona, Avatar, and Voice Packs
• Observability, Design System, Testing, and Performance
• Trust, Safety, and Privacy
• Subsystem Glossary
• V1/features.md § Psyche live-session event taxonomy and latency targets
• Hub: V1 Architecture