Lilith Metaverse · Architecture

Avatar, Animation & Spatial Audio

A focused page within the Lilith Metaverse Architecture documentation. The full map and every sibling page live in the Architecture hub.

7sections15 minread1diagram
← Back to the Architecture hubMarkdown source
On this page

V3 ("Lilith") is not a game you win; it is a place you inhabit — a multi-user metaverse where a verified yoga instructor demonstrates an asana to a room of embodied students, a Saraswati artist performs to a stadium audience, and strangers free-roam a commons. Embodiment is therefore the product, and three subsystems carry it: the avatar pipeline that turns two very different art sources (MetaHumans and VRM) into one runtime body, the animation pipeline that drives that body from canonical clips, hand-tracking, and voice, and the spatial audio layer that places every voice and music stream in 3D so the room feels like a room. Unlike V2's combat core — one provably-deterministic C++ engine — V3's embodiment stack is deliberately split across Unreal C++ (the Tier-1 native runtime under V3/ue/Source) and a tier of TypeScript domain libraries under libs/v3/ that own the import math, the retarget tables, the codec/HRTF plans, and the browser fallback. That split exists because V3 renders the same world two ways: a high-fidelity UE5 client (MetaHuman-class avatars, Lumen, Sequencer concerts) and a Tier-2 web fallback (VRM proxies, Web Audio), and the two paths must agree on skeleton, blendshapes, mix buses, and provenance even though they share no renderer.

The grounding for everything below is real and dual-located: the analytic hand-IK geometry in V3/ue/Source/V3Avatar/Public/V3HandIkSolver.h, the 60-bone Oshun→MetaHuman bind table in V3MetaHumanRetarget.cpp, the VRM 1.0 importer and blendshape semantics in libs/v3/avatar-pipeline/src/index.ts, the HRTF / ambisonic / music-sync math in libs/v3/spatial-audio/src/index.ts, and the Opus-spatializing voice gateway in V3/ue/Source/V3Voice/Private/V3VoiceRealtimeGateway.cpp. This page is the architecture-side companion for the "Avatars, Audio, and Tenant Experiences" set; the section hub is ../V3_ARCHITECTURE.md.

What ships, honestly#

The geometry, retarget tables, codec/spatialization math, and validation harnesses are real C++ and TypeScript, each backed by an automation test. On the UE side, V3Avatar ships an analytic two-bone hand-IK solver, a 60-binding MetaHuman retarget table, a four-posture state machine with IK foot/pelvis alignment, and a consent-gated Tara eyes-open policy — covered by the real automation tests V3.Avatar.OpenXRHandTracking.HandIKP95, V3.Avatar.MetaHumanRetarget.TableDrivesAsanaAndGestures, V3.Avatar.Posture.StateMachineIkAlignment, and V3.Avatar.TaraEyesOpenPolicy.SequenceDefault (all under V3/ue/Source/V3Tests/Private). V3Audio ships a Steam Audio VR profile + validator (V3.Audio.SteamAudio.VRSpatialProfile); V3Voice ships an Opus-24 kbps positional-voice gateway with equal-power panning and a mouth-to-ear latency budget (V3.Voice.RealtimeGateway.*). On the TS side, avatar-pipeline, spatial-audio, psyche-3d, and aja-pose are substantive libraries with domain-specific algorithms (VRM import + round-trip, viseme lip-sync, HRTF convolution, ambisonic decode, NTP-style music-sync clock recovery, MediaPipe/MoveNet pose estimation, asana classification), not CRUD shells.

Four honest qualifications. First, the V3Animation module is a thin registration contract — FV3AnimationModuleContract exposes only GetModuleName(), GetOwnedSurfaceTag(), and SupportsRuntimeLoad() (V3Animation/Public/V3Animation.h). The rich animation graph the monolith describes (AnimBP-per-archetype, Mover 2.0 motion matching, ChooserTable asana selection, Cascadeur/Live Link mocap) is spec-level: the retarget table and asana/gesture catalog are implemented (in V3Avatar and avatar-pipeline), but the AnimBP assets and motion-matching database are content, not code in this repo. Second, libs/v3/isis-motion is a descriptor-only package — its src/ holds a single index.ts capability descriptor with no implementation; treat it as a placeholder for the motion-authoring service, not a shipped engine. Third, the MetaHuman master assets and their hand-authored VRM proxies are content/art: the dual-authoring orchestration and likeness-drift gate are real TS (avatar-pipeline/src/premium/), but the avatars themselves are produced outside the repo. Fourth, the voice gateway's latency probes are deterministic budget-model harnesses, not live network measurements (detailed below), and Steam Audio / Resonance DSP is engine-plugin-gated — the code validates and configures the runtime via honest fail-loud seams (IsResonanceAudioRuntimeAvailable() is literally a FModuleManager::ModuleExists check) but the convolution itself runs in the plugin. The sections below say where each claim is backed.

The avatar pipeline: two sources, one body#

V3 accepts avatars from two worlds and normalizes both to a single runtime representation: the Oshun 60-bone skeleton plus a shared blendshape vocabulary. The canonical bone list is authored in two places that must agree — OSHUN_60_BONE_NAMES in libs/v3/avatar-pipeline/src/index.ts (60 entries, root → pelvis → spine01/02/chest → arms/hands/legs + finger chains + IK targets) and the matching BuildOshun60Table() in V3/ue/Source/V3Avatar/Private/V3MetaHumanRetarget.cpp, which binds each Oshun bone to its MetaHuman counterpart (spine01 → spine_01, leftHand → hand_l, leftIndexProximal → index_01_l, …). The UE table is not a flat alias map: each FV3MetaHumanRetargetBinding carries a kind — Direct, Corrective (e.g. clavicles get a ±0.5° rotation offset; facial and finger bones are corrective), or IK (knee/elbow targets) — and ValidateReferenceMetaHuman() asserts exactly 60 bindings with a max angular offset ≤ 1.0° before declaring bNoVisibleArtifacts. That same table also ships the canonical content catalogs the animation layer indexes against: CanonicalAsanaLibrary() (Tadasana, Virabhadrasana II, Adho Mukha Svanasana, Balasana, Padmasana, Savasana) and CanonicalGestureCatalog() (Namaste, Jnana mudra, Applause, Snap, Stage Wave, Consent Raise-Hand).

flowchart TD VRM[VRM 1.0 source<br/>RPM / VRoid / gallery] -->|importVrm1Document| IMP[ImportedVrm1Avatar] MH[MetaHuman source<br/>MH Creator + MH Animator] --> MHM[MetaHuman master mesh] IMP -->|createOshun60RetargetTable| SK[Oshun 60-bone skeleton] MHM -->|BuildOshun60Table / IK Retargeter| SK IMP -->|createOshunBlendshapeMap| BS[Blendshape semantics<br/>viseme / emotion / gaze / brow] SK --> T1[Tier-1 UE5 client<br/>MetaHuman + V3Avatar IK] SK --> T2[Tier-2 web fallback<br/>VRM proxy ≤ 80K tris] BS --> PSY[psyche-3d lip-sync<br/>+ expression] PSY --> T1 PSY --> T2

VRM 1.0 — the importer and its round-trip guarantee#

VRM is canonical for community avatars, Ready-Player-Me / VRoid imports, and the Oshun gallery. The importer is real TypeScript: importVrm1Document() and parseVrm1Json() parse the glTF + VRMC_vrm extension into an ImportedVrm1Avatar (avatar-pipeline/src/index.ts:843), createOshun60RetargetTable() (:920) produces an Oshun60RetargetTable that records missingRequiredBones, and verifyVrm1IdentityRoundTrip() / roundTripImportedVrm1Avatar() re-export the avatar and prove the bind survives serialization — identity is verified, not assumed. Reference locomotion is generated and retargeted by createOshun60ReferenceAnimations() and retargetVrmReferenceAnimation(), then scored by evaluateOshun60RetargetRegression() against the canonical pose set, mirroring the C++ table's role on the native side. The deeper retarget machinery lives in avatar-pipeline/src/retarget/vrm-to-oshun.ts (~21 KB).

Blendshapes — one facial vocabulary across renderers#

Both source types bind to the same blendshape semantics so a facial expression authored once plays everywhere: OSHUN_VISEME_NAMES, OSHUN_EMOTION_BLENDSHAPES, OSHUN_GAZE_BLENDSHAPES, and OSHUN_BROW_BLENDSHAPES (avatar-pipeline/src/index.ts:290+). createOshunBlendshapeMap() projects an imported avatar's expression bindings onto that vocabulary, and the lip-sync path — createReferenceVisemePhrase() → renderVisemePhrase() → evaluateVisemePhraseLipSync() (:1119–:1166) — renders viseme weights per frame and scores them. This vocabulary is intentionally the same one psyche-3d drives at runtime (next section), which is why a chat-tone-derived smile or a TTS-derived viseme means the same thing on a MetaHuman face and a VRM proxy face.

MetaHumans and the dual-authoring requirement#

MetaHumans are canonical for high-fidelity verified personas — every Saraswati artist's primary avatar, and any Tara instructor who opts in. The problem the architecture confronts head-on: a MetaHuman cannot render on the Tier-2 web client (UE-specific skeletal mesh, MetaHuman facial rig, strand-based Groom hair, Substrate materials — none port to glTF cleanly), and automated MetaHuman→VRM conversion produces unacceptable likeness drift. So every premium persona ships two parallel assets: a MetaHuman master (Tier-1 + Pixel Streaming) and a hand-authored VRM proxy (Tier-2, ≤ 80 K triangles, hair as cards, reduced viseme set, same costume material-slot identifiers). The orchestration of that workflow is real code in avatar-pipeline/src/premium/: dual-authoring.ts models the two-asset binding, likeness-drift.ts implements the release gate (the editorial owner reviews the proxy against the master from front / three-quarter / profile and a drift score must clear the §34 threshold), saraswati-ga-personas.ts enumerates the six GA personas, and tara-instructor-opt-in.ts models the post-GA opt-in queue. What is not in the repo is the art itself — the MetaHuman .uassets and the authored .vrm proxies are content. This is the honest line: the pipeline is real; the avatars are a content pipeline it gates.

Provenance and costume — identity travels with the body#

Every avatar carries an Isis provenance bundle: createImportedAvatarProvenanceBundle() / createGalleryAvatarProvenanceBundle() build it, attachProvenanceToImportedAvatar() binds it, and inspectAvatarProvenance() reads it back (:1788–:1850); both variants of a premium persona share one bundle id. Costume is realm-aware: AVATAR_COSTUME_SLOT_NAMES and REALM_COSTUME_SLOT_POLICIES (:382/:400) encode which slots each realm (tara / saraswati / commons) permits, lilithSafetyCostumeRuleCheck() enforces the per-realm safety rules, and swapCostumeVariantsWithoutReload() hot-swaps wardrobe without re-importing the avatar — the same material-slot identifiers on the MetaHuman and the VRM proxy are what make a per-realm wardrobe rule apply uniformly across both renderers. Identity binding (V1 user-id ↔ avatar-id) is owned by Lilith-Identity-Bridge, not this library; see ../../platform/oshun-domain-libraries.html for where these @oshun/* packages sit in the domain map.

The monolith's animation taxonomy (AnimBP-per-archetype, Mover 2.0 motion matching, ChooserTable asana selection, mocap via Vicon/OptiTrack/Move.AI/ Cascadeur, MetaHuman Animator facial capture) is the design; the parts that exist as verified code are the geometric and policy cores that any AnimBP would call into. There are three, all in V3Avatar, plus a runtime expression engine in psyche-3d.

Hand IK — measured, never hand-authored#

VR clients drive finger pose from OpenXR hand-tracking, and V3 solves it analytically rather than blending canned poses. V3HandIkSolver.h is deliberately UE-independent (plain double/Vec3, no FVector) so the geometry unit-tests outside the engine: FlexionFromPositions() derives a joint's flexion angle from captured parent/joint/child positions via the dot product; SolveTwoBoneFlexionForReach() is a law-of-cosines two-bone IK; and RetargetFlexionDegrees() maps a tracked flexion onto the avatar bone through a per-joint RigCalibration (rest offset, transfer gain, anatomical JointLimitDegrees default 95°) and clamps it. The UE wrapper V3OpenXRHandIk.cpp builds 52 Quest 3 joint bindings (BuildQuest3Bindings(), both hands, palm→tip) and a 20-sample validation set spanning an open palm (~2°) through a tight snap (~62°); ValidateQuest3HandIk() runs each captured pose through the solver, measures the angular retarget residual, and only sets bQuest3Ready when the binding count is 52, the sample count ≥ 20, and the P95 angular error ≤ 5°. The error is a computed geometric quantity — exactly the anti-pattern the V3HandIkSolver.h header comment calls out ("never a hand-authored literal"). This is what the V3.Avatar.OpenXRHandTracking.HandIKP95 test guards, and it is how mudra and prop interaction stay believable per finger.

Posture state machine — sit/kneel/lie need a prop anchor#

Contemplative practice means avatars sit on cushions, kneel in seiza, and lie on mats — postures most metaverse rigs fake. V3PostureStateMachine.cpp models four EV3AvatarPosture states (Stand, Sit, Kneel, Lie) with per-posture IK targets (pelvis offset, foot-lock tolerance ≤ 2 cm) and six bidirectional transitions with bounded blend durations and alignment-error caps. FV3AvatarPostureTarget::Validate() enforces a real invariant: a Sit or Lie posture must require a prop anchor (bRequiresPropAnchor), and a sitting transition must reference a sitting prop anchor — so an avatar can't float where a zafu cushion should be. ValidateSittingPropDrill() checks the full set (4 targets, ≥ 6 transitions) and a P95 prop-alignment of 1.6 cm against a 2.0 cm ceiling, which the V3.Avatar.Posture.StateMachineIkAlignment test asserts.

The most domain-specific piece of the avatar layer is also the smallest: V3TaraEyesOpenPolicy.cpp decides whether a Tara (instructor) avatar may close its eyes during a guided sequence. The default is eyes open — auto eye-close cues are suppressed unless there is an explicit, consent-shaped invitation, and HasExplicitInvitation() actually inspects the invitation text for permissive phrasing ("if it feels", "comfortable", "you may", "choose to close", …). Only then does Evaluate() apply EyesClosed with a full 1.0 close-blend weight; otherwise it returns EyesOpen with bSuppressAutoEyeClose and a recorded reason. Every decision carries a Reason string and is validated (FV3TaraEyesOpenPolicyModel::Validate()), so the policy fails loud rather than silently closing an instructor's eyes. This is consent encoded as a state machine, not a comment — guarded by V3.Avatar.TaraEyesOpenPolicy.SequenceDefault.

Runtime expression and lip-sync — psyche-3d#

The face is driven at runtime by libs/v3/psyche-3d. It targets 60 Hz lip-sync (PSYCHE_3D_LIPSYNC_TARGET_HZ) across web/mobile/VR surfaces, with a 15-viseme set; classifyPsyche3dVoiceFrame() maps a voice-analysis frame to a viseme, buildPsyche3dLipsyncTracks() produces per-surface blendshape tracks, and evaluatePsyche3dLipsyncQuality() / isPsyche3dLipsync60HzReady() gate it. It also resolves emotion from text: classifyPsyche3dChatTone() → PSYCHE_3D_TONE_TO_EMOTION → resolvePsyche3dExpressionStateFromChatTone(), with evaluatePsyche3dExpressionCoherence() checking tone/expression agreement, and parsePsyche3dGestureCommand() turning a chat gesture (/wave, /namaste) into one of the canonical gesture presets consistently across clients. This is the runtime consumer of the same blendshape vocabulary avatar-pipeline defines.

The practitioner-facing counterpart — grading the human's real pose from a webcam, not animating the avatar — is libs/v3/aja-pose. It runs on-device pose estimation (runtime.ts: MediaPipe Tasks-Vision pose-landmarker, MoveNet lightning/thunder fallback, ≥ 15 Hz on web + mobile), classifies against AJA_CANONICAL_30_ASANAS to a ≥ 92% per-asana accuracy bar (classifier.ts), generates alignment cues (cue-generator.ts), routes known-risk poses through a modification ladder (risk-flagger.ts), keeps raw pose on-device with Iris aggregate-only egress (privacy.ts), and emits Psyche-TTS instructor voice cues (voice-cue.ts). Aja grades the student; psyche-3d animates the avatar — two distinct embodiment loops that meet in a Tara class.

Spatial audio: voice, music, and a two-tier renderer#

Audio is what makes the room a room. The design splits cleanly by tier: Tier 1 (UE5) uses MetaSounds with Resonance Audio by default and Steam Audio as a VR opt-in; Tier 2 (web) uses Web Audio + Resonance Audio JS, with an ambisonic-to-stereo fallback for low-end CPUs. Both tiers share identical mix-bus semantics (voice / music / effects / ambience / accessibility), which is enforced in code: MixBusKey and createDefaultMixBusRuntime() in libs/v3/spatial-audio/src/index.ts define exactly those five buses for both paths.

Positional voice — the UE gateway#

Voice is Opus 24 kbps mono, 20 ms frames per speaker (OPUS_24K_MONO_CODEC in spatial-audio, and FV3VoiceRealtimeGatewayConfig in V3Voice/Public/V3Voice.h). UE clients receive directly via a WebRTC SFU (livekit-compatible-sfu, DTLS-SRTP with AEAD-AES-128-GCM); Pixel Streaming clients receive the already- mixed audio inside the H.264/AV1 video stream (spatialization done in the UE worker). The spatialization itself is real math in V3VoiceRealtimeGateway.cpp: from speaker/listener positions it derives DistanceM, AzimuthDegrees (atan2), and ElevationDegrees, then ApplyResonanceAudioSpatialization() computes an equal-power pan — LeftGain = atten · √(0.5·(1−pan)), RightGain = atten · √(0.5·(1+pan)) with atten = 1/(1 + 0.15·distance). The Resonance binding is an honest seam: IsResonanceAudioRuntimeAvailable() is FModuleManager::ModuleExists("ResonanceAudio"), and ConnectToRoom() fails loud ("ResonanceAudio runtime module is not available") rather than pretending. Honesty note: the 16-participant and regional probes (RunSixteenParticipantLatencyProbe, RunRegionalLatencyValidationProbe) are deterministic budget-model harnesses — mouth-to-ear latency is composed analytically as OpusFrameDurationMs + GatewayToSfuMs + SfuFanoutMs + RegionalRttMs/2 from synthesized regional-RTT fixtures, then percentiled against the 80 ms budget. They validate the latency model and the spatialization math, not a live network, which is the correct scope for an automation test (V3.Voice.RealtimeGateway.SfuOpusSpatializationLatency / ...RegionalLatencyP95).

Steam Audio VR profile#

Higher-fidelity VR occlusion is the V3Audio Steam Audio profile. FV3SteamAudioVrRuntimeProfile::BuildDefaultVrProfile() configures 48 kHz, 8 reflection bounces, a 30 Hz occlusion trace rate, HRTF + occlusion + reflections

  • voice + music spatialization, and an explicit voice-latency budget that sums to 70 ms (22 capture-to-SFU + 18 jitter + 12 DSP + 10 mixer + 8 output) under an 80 ms ceiling, across the launch XR backend matrix (Quest 3, Vision Pro, PSVR2, Valve Index, Vive Focus 3). Validate() rejects any profile that doesn't bind the SteamAudio plugin and HRTF dataset, isn't the VR-opt-in default, drops a spatialization feature, or blows the latency budget — and V3.Audio.SteamAudio.VRSpatialProfile asserts the validator passes. The DSP runs in the Steam Audio plugin; this code owns the contract and the budget, not the convolution.

Music sync and the Tier-2 fallback#

Stadium-tier concerts need every attendee to hear the music in lockstep. The sync is NTP-style: createMusicStreamServerTimestamp() stamps the stream, estimateMusicSyncClock() recovers the server-clock offset from a four-timestamp exchange (with round-trip-delay smoothing), mapMusicStreamTimestampToPlaybackTarget() turns that into a client playback target with a playout buffer, and simulateMusicSyncConcert() validates a 256-attendee, hour-long concert against a P99 drift budget of 25 ms (MUSIC_SYNC_P99_DRIFT_BUDGET_MS). The Tier-2 renderer is real Web Audio: createHrtfConvolutionPlan() builds an AudioWorkletNode → ConvolverNode → GainNode chain, decodeFirstOrderAmbisonicToStereo() is the low-CPU fallback, selectSpatialRendererForDevice() chooses between them per device profile, and createTier2FallbackSpatialAudioScenePlan() wires Resonance Audio JS (ResonanceAudio.Source → ResonanceAudio.Scene node chain) — the resonance-audio module surface is declared in spatial-audio/src/resonance-audio.d.ts. Scene occlusion is sampled at 5 Hz (evaluateSceneOcclusion, SCENE_OCCLUSION_SAMPLE_RATE_HZ), and accessibility is first-class: an audio-description bed (createAudioDescriptionBedPlan) and a flat-stereo spatial-audio-off mode (createSpatialAudioOffModePlan) that bypasses ambisonics for pixel-streaming / fallback / native-mobile / VR clients.

Edge cases and failure modes#

  • Hand IK degrades to zero, never to garbage. FlexionFromPositions() returns 0 for a degenerate (zero-length) bone segment, and RetargetFlexionDegrees() clamps to the anatomical joint limit — a bad capture can't hyperextend a finger.
  • Posture won't float. Validate() blocks a sitting or lying posture that lacks a prop anchor, so an avatar physically requires the cushion/mat to exist.
  • Eyes-closed needs words. The Tara policy suppresses an auto eye-close cue unless the invitation text is explicitly permissive; absent that, it records eyes-closed-auto-cue-suppressed-explicit-invitation-required and keeps eyes open.
  • Voice fails loud. No Resonance module → ConnectToRoom() returns false with a reason; wrong participant count → an explicit "expected N, got M"; a non-Resonance spatializer is rejected ("V3Voice must render positional voice with Resonance Audio").
  • Music sync is bounded, not best-effort. The concert simulation asserts a P99 drift ≤ 25 ms; the clock estimator smooths round-trip delay so a single jittery exchange can't yank the offset.
  • Spec vs. content seam. The V3Animation module and isis-motion package are intentionally thin; the AnimBPs, motion-matching database, MetaHuman masters, and VRM proxies they presuppose are authored content. Reading those modules expecting a motion engine will mislead — the engine-side logic lives in V3Avatar (IK, retarget, posture) and psyche-3d/aja-pose (expression, grading).

How it connects#

The avatar/animation/audio stack is the embodiment substrate the rest of V3 reads. The Oshun-60 skeleton, blendshape map, and MetaHuman/VRM bodies are rendered by the Tier-1 UE5 client and its Pixel- Streaming workers, and proxied to the browser through the tier-routing path. The voice gateway and music-sync clock ride the wire protocol described in netcode, protocol, and physics, whose per-mode tick targets (60 Hz expression, ≤ 25 ms music sync for Saraswati performers) are exactly what psyche-3d and the music-sync simulator are sized against. The two tenant experiences consume this stack directly: Saraswati stage pipeline drives the premium MetaHuman performers, concert costume packs, and stadium music sync, while Tara classes, Aja, and the Commons drive the instructor avatars, the Aja webcam pose-grading loop, asana retargeting, and the eyes-open consent policy. Provenance and identity binding reach into the V1 domain libraries catalogued in ../../platform/oshun-domain-libraries.html, and the module topology (V3Avatar / V3Animation / V3Audio / V3Voice and the libs/v3/* packages) is mapped in the subsystem glossary and layout.