Psyche Real-Time Runtime

Psyche is the V1 real-time embodiment substrate: it owns the contract for a live text, voice, and avatar session — the session envelope, the event model, turn-taking, interruption, reconnect, transcript sync, lip-sync, expression coherence, latency budgets, quality thresholds, fallback routing, crisis-frame continuity, and diagnostics. It serves every customer-facing surface that needs to embody a persona in real time (the assistant voice mode, a Metis tutor session, a Living Scene), and it sits underneath the persona/policy substrate (Lilith Persona Policy) and alongside identity and memory (Iris Memory and Identity). For the substrate's place in the platform see the companion Psyche — Real-Time Runtime Substrate.

What is real today, and what is spec-only#

Psyche's contract and logic layer is real and rich, and it is deliberately narrow. The package @oshun/embodiment-psyche (v0.1.0, type: module, with main/types pointing at ./src/index.ts source — there is no compiled dist) is built as pure data + pure functions with no IO, no rendering, and no transport. Nearly every module carries the literal fileoverview line Pure data + pure functions. No IO. (session-events.ts, session-envelope.ts, latency-dashboard.ts, backpressure.ts, fallback-routing.ts, reconnect-behavior.ts, avatar-sync.ts — avatar-sync.ts adds No IO, no rendering.). The adapter code (src/adapter.ts, src/canonical-adapter.ts) imports no fetch, http, ws, prisma, or redis: it wraps an injected low-level adapter and never reaches the network itself. The library defines the shape of correctness; the runtime that binds it to a wire, a media pipeline, and a provider is a separate concern.

What is spec-only / aspirational is exactly that binding: the live wire/transport, the actual ASR/TTS/avatar provider integrations, and the operator surfaces that consume the contracts. Those service-backing trees exist with substantive content — services/psyche/{orchestrator, avatar-engine, behavior-engine, conferencing, voice-engine, tavus-integration, persona-service, perception-engine, …}, infrastructure/psyche/{docker, kubernetes, terraform}, libs/contracts/psyche/src, and apps/psyche/admin (only admin is present) — plus the much larger libs/psyche/* substrate tree (~136 subdirs including tavus-*, avatar-*, memory-*, viseme-generator, voice-synthesis, speech-recognition), which is not the same library as the embodiment-psyche adapter contract. The completeness audit rates the psyche-tutor-live-session-to-graded-record walkthrough as partial, with the live-voice envelope, library write, and operator review lane noted as unbuilt surfaces. So: the contract+logic layer is real; the end-to-end live voice/avatar session is partially built. This page describes the contract honestly and flags where the runtime is still aspirational.

The canonical entry point is createCanonicalPsycheEmbodimentAdapter in src/canonical-adapter.ts, not adapter.ts. The factory takes an { apiAdapter } (a PsycheEmbodimentApiAdapter) and returns a PsycheEmbodimentAdapter that layers contract descriptors, availability, capability resolution, and session planning (buildPsycheSessionPlan, which refuses to start when plan.approved is false) over the injected adapter — without performing any IO of its own. The session machinery lives across the 23 modules re-exported by src/index.ts: canonical-adapter, session-envelope, session-events, voice-orchestration, reconnect-behavior, transcript-sync, multimodal-state, latency-dashboard, backpressure, quality-thresholds, fallback-routing, session-audit, session-diagnostics, avatar-sync, emotion-modulation, screen-context, crisis-frame, provider-failover, plus types, embodiment-model, adapter, session-trace, and events/index.

Session Envelope and Event Model#

Session envelope#

The session envelope (session-envelope.ts, interface PsycheSessionEnvelope) is a rich, validated record — not a loose bag of fields. It carries the session identity and mode, locale and region, capabilities, entitlement class, the Lilith policy version, the transport contract, the lifecycle state, the continuity block (memory scope, disclosure, grounding mode), and an integrity fingerprint.

The capability list (PSYCHE_SESSION_CAPABILITIES) is exactly the set of wire event kinds the session is allowed to emit plus the four richer capabilities lip-sync, emotional-modulation, screen-context, and translation. This is why the envelope's capabilities and the event taxonomy line up: a session that cannot emit viseme-stream is, by construction, a session that did not negotiate the lip-sync capability.

Beyond the static record, the envelope module ships real operations worth surfacing: a lifecycle transition graph (PSYCHE_SESSION_LIFECYCLE_TRANSITIONS, enforced by transitionPsycheSessionEnvelope, where the terminal states closed/failed/timed-out accept no further transitions), modality negotiation (negotiatePsycheSessionModalities), persona-switch planning (planPsycheSessionPersonaSwitch), conservative per-transport-tier latency budgets, and the fingerprinting used to detect envelope tampering or drift.

Event types — the dual taxonomy#

The runtime speaks two event vocabularies, both fully defined in session-events.ts. The 15 canonical V1 wire event types (PSYCHE_V1_SESSION_EVENT_TYPES, line ~115) are an exact match for the list in ../features.md:

text-token-stream  asr-partial  asr-final  tts-chunk  avatar-frame
viseme-stream  expression-update  turn-complete  interruption  reconnect
transcript-sync  error  kill-switch  fallback-engaged  policy-intervention

Alongside these, the library also ships the legacy dotted protocol (PSYCHE_LEGACY_SESSION_EVENT_KINDS) and two server turn events (PSYCHE_SERVER_TURN_EVENT_KINDS = ['turn-start', 'turn-end']). The union of all three is PSYCHE_SESSION_EVENT_KINDS, and every kind is reachable through the discriminated union type PsycheSessionEvent. The legacy family is what the server turn reducer and lower layers emit:

Every event — legacy or V1 — is a PsycheSessionEventBase carrying the same spine: eventId, kind, sessionId, a per-session traceId, a timing record (ingress/egress timestamps, optional provider attribution, and retryAttempts), a monotonic sequence, an emittedAt ISO timestamp, an actor (PsycheSessionEventActor), the typed payload, and optional correlationId / causeEventId links (so turn.claim → turn.grant chains remain traceable).

Gap-free, monotonic event stream#

The event stream is append-only with enforced ordering. The stream record (PsycheSessionEventStream, version PSYCHE_SESSION_EVENT_STREAM_VERSION = 1) holds a shared traceId and a nextSequence counter. appendPsycheSessionEvent rejects any event that breaks the invariants — these are real guard clauses, not narration:

• event.sessionId must equal the stream's sessionId;
• event.traceId must equal the stream's traceId;
• event.sequence must equal the stream's nextSequence (no gaps, no reordering);
• event.emittedAt must not precede the previous event's emittedAt (non-decreasing wall clock).

Only on passing all four does the function return a new stream with the event appended, lastUpdatedAt advanced, and nextSequence incremented. This is what makes replay and audit trustworthy: a captured stream is provably gap-free and monotonic.

Turn-taking — the server turn reducer#

Turn-taking is server-mediated and backed by a real reducer, not just described narratively. createPsycheServerTurnState builds a PsycheServerTurnState (with sessionId, traceId, activeTurnId, activeSpeakerId, activeTurnStartedAt, a phase, the partialTtsFadeOutMs, and nextSequence), and applyPsycheServerTurnCommand advances it. The phase is the type PsycheServerTurnPhase = 'idle' | 'active' | 'interrupted' | 'complete', and the commands are a small discriminated union:

The barge-in policy the doc describes — when the user starts speaking, partial TTS is faded out within a configurable number of milliseconds — is the constant PSYCHE_DEFAULT_PARTIAL_TTS_FADE_OUT_MS = 180 (session-events.ts, line ~768). Both interruption and system-initiated-barge-in accept a per-command partialTtsFadeOutMs so the default can be tuned per session, and the reducer normalizes/clamps the value before applying it.

Reconnect#

Reconnect (reconnect-behavior.ts) is interrupt-tolerant and built on a durable session id, the last acknowledged event sequence, and a replay buffer. The reconnect logic restores session state, replays the partial-TTS that was in-flight at the cut, and reconciles the transcript so the resumed session does not duplicate or drop tokens. The reconnect path is itself pure (No IO, no timers — the caller runs the schedule): it computes the backoff plan and the replay window; the runtime executes them. Reconnect attempts are capped by the transport contract's maxReconnectAttempts, and exhaustion is a first-class audited outcome.

Multimodal sync#

Multimodal coherence (multimodal-state.ts, avatar-sync.ts) aligns three clocks: per-token timing for text, viseme alignment for audio, and expression events for semantic content. Lip-sync is concrete, not abstract: avatar-sync.ts ships 11 visemes (PSYCHE_AVATAR_VISEMES = [rest, ah, ee, oh, oo, mb, fv, ss, th, eh, ay]) and a PSYCHE_AVATAR_VISEME_TABLE mapping the 39 ARPABET phonemes into those 11 (e.g. AA/AH/AO → ah, IH/IY/Y → ee, B/M/P → mb, F/V → fv, S/Z/SH/ZH/CH/JH → ss, SIL → rest; unknown phoneme ids resolve to rest). interpolatePsycheAvatarPose linearly blends between poses, buildPsycheAvatarPhonemeTimeline orders phonemes into a timeline, and assessPsycheAvatarAlignment measures the audio-vs-pose gap and reports a status of aligned / drifting / desynced (PSYCHE_AVATAR_ALIGNMENT_STATUSES) so a drifting avatar can trigger re-render or fallback. Screen-context awareness (screen-context.ts) — current page and current artifact — is surfaced only where consented, in keeping with the disclosure baseline on the envelope.

Latency Budgets and Quality Thresholds#

Every latency target in ../features.md is a literal constant in latency-dashboard.ts (lines ~68–84) — these are exact matches, not approximations:

Observations are typed PsycheLatencySamples (kind, valueMs, observedAtMs) with kinds including avatar-frame, voice-first-audio, and first-token. assessPsycheLatencyAgainstBudget compares measured percentiles to the envelope's budget and returns a PsycheLatencyBudgetStatus of ok / warning / exceeded (PSYCHE_LATENCY_BUDGET_STATUSES) for the round-trip, the pipeline, and overall — the raw material for the quality dashboards (p50/p95/p99 per region, per provider, per persona) and SLO-breach alerts the doc promises.

Backpressure — admission control and shed planning#

Backpressure (backpressure.ts) is more than "apply pressure when a provider degrades" — it is two real mechanisms plus a provider response plan:

1. Admission control. decidePsycheSessionAdmission returns a typed decision (PSYCHE_SESSION_ADMISSION_DECISIONS) weighing the request against capacity and a priority (PSYCHE_SESSION_ADMISSION_PRIORITIES), so a saturated region can refuse or queue a new session rather than degrade every live one.
2. Shed planning. planPsycheBackpressureShed selects which running sessions to degrade and how, using the typed actions in PSYCHE_BACKPRESSURE_SHED_ACTIONS. The aggregate health rolls up to a PSYCHE_BACKPRESSURE_STATUSES value of healthy / elevated / critical.
3. Provider response. When a downstream provider degrades, planPsycheProviderBackpressureResponse emits a plan drawn from PSYCHE_PROVIDER_BACKPRESSURE_ACTIONS (asr-throttle, model-streaming-throttle, show-thinking-indicator, fallback-to-text).

The defaults live in PSYCHE_PROVIDER_BACKPRESSURE_DEFAULT_POLICY:

{
  "thinkingIndicatorDeadlineMs": 200, // visible "thinking" cue must be scheduled within this; clamped to <= 200ms
  "fallbackToTextAfterMs": 2500, // how long a degraded provider may persist before falling back to text
  "asrThrottleRatio": 0.5, // fraction of normal ASR intake allowed under backpressure
  "modelStreamingThrottleRatio": 0.5, // fraction of normal model-token streaming allowed under backpressure
}

The thinkingIndicatorDeadlineMs of 200 is exactly the doc's "user-visible 'thinking' indicator within 200 ms" — and the code clamps any override to ≤ 200 ms via Math.min, so the indicator can be made faster but never slower.

Quality thresholds#

Quality thresholds (quality-thresholds.ts) cover lip-sync alignment per locale, expression coherence, and voice naturalness, each compared to a baseline. A below-threshold reading is what triggers a re-render or a fallback step; the provider backpressure plan and the latency-budget status feed the same decision-making so quality regressions and latency regressions are handled by one coherent escalation path rather than two divergent ones.

Fallback Chain#

The fallback chain is grounded in fallback-routing.ts. The ordered modes are PSYCHE_FALLBACK_MODES = ['avatar', 'voice', 'text', 'unavailable'], and the chain degrades in exactly the order the doc states — avatar + voice → voice → text — with unavailable as the terminal rung. resolvePsycheFallbackMode ranks the requested mode against what is actually attainable (via a capability probe, buildPsycheCapabilityProbe) and never silently upgrades.

Fallback is triggered by the set PSYCHE_FALLBACK_TRIGGERS (session-events.ts, line ~360): provider-failure, quality, latency, kill-switch, policy, and accessibility — a one-to-one match for the doc's "provider failure, quality-threshold breach, latency-budget breach, kill-switch, policy intervention, accessibility request." Each step is user-visible — describePsycheFallbackNotice produces the per-stage disclosure copy — and the user can decline a step.

buildPsycheFallbackCascadePlan composes the full descent, and planPsycheFallbackRenegotiation rewrites the envelope so that persona memory and grounding state are preserved across mode transitions and transcript continuity is maintained. Recovery is governed by PSYCHE_FALLBACK_UPGRADE_ACTIONS = ['hold', 'offer-upgrade', 'attempt-upgrade'] via planPsycheFallbackUpgradeAttempt, so an automatic upgrade is attempted only after a stable interval, and a PsycheFallbackUserReengagementRequest lets the user re-engage a richer mode on their own terms.

Kill switch#

The kill switch is the hard stop above ordinary fallback. Its scopes are PSYCHE_KILL_SWITCH_SCOPES = ['session', 'tenant', 'region', 'global'], and when fired it forces one of PSYCHE_KILL_SWITCH_FALLBACK_MODES = ['voice', 'text', 'closed']. A kill-switch event is one of the 15 V1 wire types, and a kill-switch.fired audit record (see below) is written every time it engages, at whatever scope.

Diagnostics, Replay, and Audit#

Every event is tagged with a per-session traceId for end-to-end correlation, and each event's timing block carries ingress/egress timestamps, provider attribution, and retryAttempts — so a slow turn can be attributed to the exact hop and provider that caused it.

session-diagnostics.ts is the operator's postmortem toolkit and is real code, not a promise: buildPsycheSessionPostmortemBundle assembles a session bundle, replayPsycheSessionLifecycle walks the audit trail and reconstructs the lifecycle-state history, and replayPsycheSessionAgainstFixture (operating over a PsycheSessionOperatorReplayFixture) replays a captured event fixture through the canonical builders to reconstruct the session and report PsycheSessionOperatorReplayIssues — exactly the "operator can replay a session against fixtures with full event reconstruction" the doc describes. (The operator UI that drives this is one of the noted unbuilt surfaces; the engine is here.)

The audit trail (session-audit.ts) records the safety- and continuity-relevant events with their own kind taxonomy (PsycheSessionAuditEventKind), including reconnect.attempted / reconnect.succeeded / reconnect.failed, fallback.renegotiated / fallback.engaged, persona.break, lilith.intervention, and kill-switch.fired. So every persona break, every Lilith policy intervention, every kill-switch firing, every fallback engagement, and every reconnect is logged — auditable after the fact and reconstructable via replay.

Continuity Tests#

The continuity guarantees the doc lists are each backed by a real module:

• Reconnect under network loss — reconnect-behavior.ts resumes mid-turn within the replay window (durable session id + last-acked sequence + replay buffer; partial-TTS replay; transcript reconciliation).
• Device handoff mid-turn — modeled as a reconnect.handoff event so the session can be transferred to another device with state preserved.
• Persona switch mid-session — planPsycheSessionPersonaSwitch plans a graceful transition that keeps conversation context and the Iris memory scope intact.
• Provider failover mid-turn — provider-failover.ts degrades seamlessly, emitting user-visible disclosure, preserving transcript continuity, and writing an audit trace.

Crisis-frame entry mid-session#

Crisis-frame entry is a first-class implemented module (crisis-frame.ts), not narration. enterPsycheCrisisFrame rewrites the live envelope to a safety posture and emits the right events and audit records in one atomic step:

• continuity.memoryScope → 'off' (memory writes suspended);
• embodiment.syntheticVoice → false and syntheticAvatar → false, with the voice/avatar pack ids cleared (synthesis halted);
• lifecycleState → 'degraded';
• a policy-intervention wire event whose actor is the system actor { kind: 'system', id: 'lilith' };
• two audit events — persona.break and lilith.intervention — both attributed to the same Lilith system actor, with the actions payload PSYCHE_CRISIS_FRAME_AUDIT_ACTIONS = ['break-persona', 'halt-synthesis', 'suspend-memory-writes'];
• a PsycheCrisisFrameSynthesisHalt plan (which modalities, pipeline services, and capabilities to remove) and a PsycheCrisisFrameMemorySuspension record (capturing the prior memory scope so it can be reasoned about), plus a fixed user-visible disclosure line.

This is the runtime expression of Lilith Persona Policy: the moment Lilith determines a session must break for safety, Psyche enforces the triple action — persona break, synthesis halt, memory-write suspension — and leaves an audit trail that names Lilith as the actor.

Living Scenes integration#

Psyche carries Living Scene sessions over the same envelope/event/reconnect machinery used for text/voice/avatar. The event model is extended with scene-specific kinds in events/scene-events.ts (SCENE_EVENT_KINDS): scene.segment-start, scene.segment-end, scene.transition-start, scene.transition-end, scene.live-direction-applied, scene.policy-intervention, scene.crisis-frame, and scene.fallback-engaged — validated by validateSceneEvent and stream-checked by verifyEventStream.

The frame-stream channel is real backpressure code (events/frame-stream.ts): it multiplexes scene frame deliveries and applies a graduated BackpressureLevel of nominal → drop-non-keyframes → halve-frame-rate → blocked as a degraded provider falls behind (evaluateBackpressure, stepFrameStream, consumeFrame). Cue Plan progress is preserved across reconnect alongside the transcript and the Iris memory scope — events/cue-plan-replay.ts ships validateCuePlan and computeReplay to reconstruct cue progress. See Live Direction, Conductor Runtime, and Blend Kernel for the scene runtime that drives these events.

How a turn flows (data-flow walkthrough)#

Putting the pieces together, a single voice turn moves through the contract like this — every step naming a real symbol:

1. A PsycheSessionEnvelope is built and validated; its capabilities, mode, transport tier, entitlementClass, lilithPolicyVersion, and continuity.memoryScope are fixed, and a fingerprint is computed.
2. The user starts speaking. The server turn reducer is in idle; applyPsycheServerTurnCommand with turn-start moves it to active and appends a turn-start event (sequence checked by appendPsycheSessionEvent).
3. ASR streams asr-partial events, then an asr-final. The ASR final-text latency is sampled and assessed against PSYCHE_ASR_FINAL_TEXT_LATENCY_TARGETS (p50 ≤ 150 ms).
4. The model streams text-token-stream; first-token latency is checked against PSYCHE_FIRST_TOKEN_LATENCY_TARGETS (p50 ≤ 350 ms). If the provider lags, planPsycheProviderBackpressureResponse schedules a show-thinking-indicator within thinkingIndicatorDeadlineMs (≤ 200 ms).
5. TTS streams tts-chunk; voice first-audio latency is checked against PSYCHE_VOICE_FIRST_AUDIO_LATENCY_TARGETS (p50 ≤ 500 ms). In avatar mode, viseme-stream and avatar-frame events are aligned by assessPsycheAvatarAlignment, with avatar-frame latency held to ≤ 80 ms.
6. If the user barges in, a system-initiated-barge-in/interruption command fades out partial TTS within PSYCHE_DEFAULT_PARTIAL_TTS_FADE_OUT_MS (180 ms) and moves the reducer to interrupted.
7. The turn closes with a turn-end command → turn-complete event; the reducer returns to idle/complete. If a provider failed or quality fell below baseline, buildPsycheFallbackCascadePlan degrades the mode with a user-visible notice; if Lilith intervenes, enterPsycheCrisisFrame halts synthesis, suspends memory writes, and emits the policy/audit events.

Throughout, the transport that carries these events and the providers that produce the audio/avatar bytes are outside this library — the contract says what is correct; the (partially built) services under services/psyche/* and libs/psyche/* are what make it move.

Related#

• Psyche — Real-Time Runtime Substrate — the substrate companion page.
• Lilith Persona Policy — the policy substrate that fires crisis-frame entry and kill switches.
• Iris Memory and Identity — the memory scope the envelope binds and the crisis frame suspends.
• Persona, Avatar, and Voice Packs — the embodiment assets a session binds.
• Assistant Experience — the primary surface that drives a live Psyche session.
• Domain Metis — Education and Tutoring — the tutor surface behind the psyche-tutor-live-session-to-graded-record walkthrough.
• Living Scenes — Concept and Customer Promise and Live Direction, Conductor Runtime, and Blend Kernel — the scene runtime that rides the same envelope/event machinery.
• Subsystem Glossary and the feature hub ../features.md.