aja library

Runtime animation-blending library for Lilith avatars (libs/aja/animation-blending/src). It provides idle blending (idle-blending.ts), clip transitions (clip-transitions.ts), additive layers (additive-layers.ts), and body-part masking (body-masks.ts) over a real vector/quaternion math kernel in blend-utils.ts (slerp, additive-difference transforms, 1D/2D blend-weight calculation, ping-pong/wrap time utilities). This is the playback-side animation mixer, not a capture stage.

The teacher-avatar integration layer (libs/aja/avatar-integration/src) that connects motion-capture data to instructor avatars. It ships concrete pipelines for yoga (yoga-pipeline.ts, with breath visualization), fitness (fitness-pipeline.ts, rep counting / form feedback), and meditation (meditation-pipeline.ts), plus a custom-avatar.ts path for user-imported characters, over a large shared type surface (avatar identity, skeleton, blend shapes, pipeline stages). It is the glue between captured motion and a specific on-screen instructor.

GPU-oriented batch-inference scheduling for pose/motion analysis (libs/aja/batch-inference/src). It implements frame batching (frame-batcher.ts), multi-video batching (video-batcher.ts), a latency/memory-driven dynamic-batch-sizer.ts, and GPU memory pooling / OOM-recovery (memory-manager.ts), wired together by batch-scheduler.ts with an injectable inference callback. It models the batching/scheduling control plane; the actual model call is supplied by the caller via setInferenceCallback.

Blend-shape / morph-target retargeting for faces and soft-body deformation (libs/aja/blend-shape-retargeting/src). It covers facial expression transfer (facial-expression.ts, with ARKit/FACS/viseme vocabularies in types.ts), muscle deformation (muscle-deformation.ts), secondary motion (secondary-motion.ts), and cloth simulation (cloth-simulation.ts), composed in pipeline.ts. It is the face/morph counterpart to the skeletal-retargeting band.

Video-to-CG character-replacement primitives (libs/aja/cg-replacement/src): actor segmentation, 3D pose extraction, facial-rig/landmark estimation, lighting-aware rendering, compositing, shadow/reflection synthesis, and temporal consistency. The work is concentrated in video-to-cg-pipeline.ts over a very broad type surface in types.ts (FACS action units, lip-sync frames, depth samples, estimated lights). It extends the motion core toward replacing an actor in footage with a CG character rather than only animating an avatar.

Activity-specific motion-analysis pipelines (libs/aja/domain-motion-pipelines/src) for yoga, fitness, dance, and martial arts. The yoga pipeline is the deepest — asana detection (ASANA_DATABASE), alignment analysis, breath/drishti/bandha detection, meditation-state and pranayama recognition, and sequence tracking — with sibling fitness-pipeline.ts, dance-pipeline.ts, and martial-arts-pipeline.ts. metis-relationship.ts types how each pipeline maps to a Metis embodied-instruction moment (movement→lesson-path, demonstration→study-pack, coaching→tutoring), making this the cross-domain bridge to the learning side.

A focused (single-file, ~370 LOC) film/VFX delivery builder (libs/aja/film-pipeline/src/index.ts) built on @aja/motion-formats. Given a clip and delivery tier (editorial / vfx-review / final-vfx / archive), it builds per-format deliverables (USD/Alembic/FBX/BVH/glb with correct MIME types and file naming), validates them (frame/rate/timecode/format constraints, with final-VFX requiring USD/Alembic/FBX), computes a 0–1 deliveryScore, and emits a versioned JSON manifest (aja-film-pipeline/v1). Small but real — a deterministic package/validator, not a stub.

The largest library in the area (libs/aja/fitness-animation/src, ~28K LOC across 40 modules): a full skeletal-animation engine for fitness motion. It spans an extended skeleton (65+ joints, skeletal/), physics-based dynamics with balance / ground-contact / impact / resistance (physics/), IK solvers (FABRIK, CCD, Jacobian, analytical in kinematics/ik-solver.ts), FK and procedural noise, mocap import/cleaning/retargeting (mocap/, including BVH/C3D/FBX parsers), muscle simulation (muscle/), secondary motion, fatigue modeling (fatigue/), and pose estimation/scoring (pose-estimation/, MediaPipe/MoveNet adapters). A broad, real animation-and-analysis stack.

Text-to-motion and generative-animation primitives (libs/aja/generative-animation/src, mostly generative-animation.ts). It provides deterministic diffusion-plan conditioning (DiffusionDenoisingStep), physics-aware skeleton generation with ballistic trajectories and contact constraints, style/interaction controls, in-betweening, motion looping, natural-language motion editing, and DCC export artifacts/payloads. The diffusion is a deterministic plan/condition model (not a trained sampler), with the focus on the physics and constraint synthesis around it.

Inference-model optimization (libs/aja/model-optimization/src): TensorRT compilation (tensorrt-compiler.ts), ONNX graph optimization (onnx-optimizer.ts), mixed-precision planning (mixed-precision.ts, FP16/BF16/ INT8/INT4), and operator fusion (operator-fusion.ts). It models the optimization workflow — precision configs, fusion patterns, INT8 calibration — but the TensorRT engine build is simulated (in-source: "Build the engine (simulated — in production would use native bindings)"); there are no native TensorRT/ONNX runtime calls here.

Validation and benchmarking infrastructure (libs/aja/motion-validation/src, ~14K LOC). It defines standard skeletons and ground-truth dataset management (ground-truth.ts — Human3.6M / CMU / COCO), evaluation metrics (metrics.ts — MPJPE/PA-MPJPE/N-MPJPE/PCK, velocity/acceleration/jerk, anatomical plausibility, footskate), plus regression, stress, edge-case, and human-evaluation harnesses. The deeper testing-rig counterpart to motion-quality's per-clip scoring.

Multi-camera 3D reconstruction (libs/aja/multi-view-reconstruction/src). It implements camera calibration (calibration.ts — intrinsics/extrinsics), 3D triangulation from multiple views (triangulation.ts), view synchronization (synchronization.ts — audio/visual/timecode), multi-view pose fusion (fusion.ts), and volumetric capture (volumetric.ts — point clouds, TSDF, mesh extraction). A real multi-view geometry stack on the capture side.

Monocular 2D→3D pose lifting (libs/aja/pose-lifting/src). It implements lifting networks (lifting-networks.ts — VideoPose3D/PoseFormer/MixSTE/MotionBERT/ MHFormer architectures), depth-estimation integration (depth-estimation.ts — MiDaS/ZoeDepth/Depth-Anything), camera-intrinsics estimation (camera-estimation.ts), bone-length optimization (bone-optimization.ts), and anthropometric validation, fronted by a createPoseLiftingService orchestrator. Real geometry and bone/anthropometric optimization; the learned lifting backbones are the model seam the service drives.

Proportional adaptation for retargeting between differently-proportioned skeletons (libs/aja/proportional-adaptation/src). It implements limb-length scaling (limb-scaling.ts), reach adjustment so a character can still reach targets (reach-adjustment.ts), ground-contact adaptation to prevent foot penetration/sliding (ground-contact.ts), and self-collision avoidance (collision-avoidance.ts), composed by pipeline.ts / runProportionalAdaptation. A real geometric adaptation stage in the retargeting band.

Semantics-aware retargeting that preserves motion intent (libs/aja/semantic-retargeting/src). It performs action recognition (action-recognition.ts — locomotion/manipulation/gesture/combat), semantic feature extraction (semantic-features.ts — energy/tempo/spatial extent), context-aware joint prioritization (joint-priorities.ts), intent preservation with verification (intent-preservation.ts), and an extensible rules-engine.ts, composed in pipeline.ts. It sits above geometric retargeting to keep the meaning of a motion intact.

Video chunking for distributed processing (libs/aja/video-chunking/src). It performs temporal splitting (video-splitter.ts), overlap handling for boundary continuity (overlap-handler.ts), result merging (result-merger.ts), and seamless reconstruction (reconstructor.ts) over chunk/time/frame-range types. It is the scatter half of the distributed pipeline, paired with result-aggregation on the gather side.

Consent tracking for data usage, uploads, and processing (libs/aja/consent-management/src). It provides a core ConsentManager with InMemoryConsentStorage, plus specialized managers for upload consent (upload-consent.ts), verification (consent-verification.ts, email/SMS), and withdrawal (withdrawal-manager.ts), with audit logging and legal-basis tracking in types.ts. The storage seam is pluggable; the in-memory implementation is the default backing.

Upload scanning and moderation workflow (libs/aja/content-moderation/src). UploadScanner composes pluggable ContentDetectors — NsfwDetector, ViolenceDetector, CopyrightDetector, PolicyViolationDetector — and moderation-workflow.ts adds a ModerationQueue, AppealManager, BanManager, and AutomatedModerator. The detector interface is the seam where a real classifier would plug in; the workflow/queue/appeal/ban state machine around it is fully implemented.

Data-retention management with GDPR-style compliance (libs/aja/data-retention/src). It includes a RetentionPolicyManager with conditions/actions and default policies, a DeletionManager with scheduling / grace periods / legal holds (over an InMemoryRetentionStorage seam), multi-format DataExportManager (JSON/CSV/XML), and a right-to-be-forgotten.ts implementation (GDPR Article 17) that issues erasure certificates. Compliance logic is real; persistence is a pluggable storage interface.

Visual privacy protection for mocap assets (libs/aja/privacy-protection/src). It provides face detection/tracking (face-detection.ts), multiple anonymization techniques (face-anonymizer.ts — blur, pixelate, mask, silhouette), identity-based selective anonymization (identity-manager.ts), and policy management for consistent rules. The detection/identity stages are the model seams; the anonymization transforms and policy logic are concrete.

Mocap format conversion (libs/aja/motion-formats/src; package @aja/motion-formats) and a foundational dependency for the area. It provides import/export for BVH (bvh.ts), FBX (fbx.ts), glTF/GLB (gltf.ts), USD (usd.ts), and Alembic (alembic.ts), plus a searchable clip-database.ts, a generic mocap-import.ts, and quality/validation helpers. Its AnimationClip / MotionFormat types are consumed across the area (e.g. by film-pipeline).

Mocap cleanup and enhancement (libs/aja/motion-processing/src; package @aja/motion-processing). It implements noise reduction with real signal processing (noise-reduction.ts — FFT/IFFT, power spectrum, dominant-frequency analysis), foot-sliding correction (foot-sliding.ts), physics-based cleanup (physics-cleanup.ts — CoM smoothing, balance validation), retiming (retiming.ts), gap filling (gap-filling.ts), and segmentation (segmentation.ts). A real DSP-grade cleanup stage between capture and retargeting.

Motion-quality assessment (libs/aja/motion-quality/src; package @aja/motion-quality). ground-truth-comparison.ts implements real metrics — MPJPE, PA-MPJPE (Procrustes-aligned), PCK, AUC with Euclidean joint distances — alongside basic metrics (quality-metrics.ts: jitter, foot-sliding, bone-length, joint angles), perceptual metrics (perceptual-metrics.ts), and an automated QA pipeline (qa-pipeline.ts) that validates/rejects/reports. The metrics compute against actual joint data, not placeholders.

Neural motion retargeting (libs/aja/neural-retargeting/src; package @aja/neural-retargeting). networks/ implements real TS tensor ops and layers (layers.ts — Float32Array tensors, graph-conv weights) for Skeleton-Aware Networks (san.ts), Neural Kinematic Networks (nkn.ts), a transformer (transformer.ts), and AdaIN style transfer (style-transfer.ts). inference.ts runs through an injectable ONNX Runtime seam and falls back to linear-interpolation retargeting when no model is loaded — real network code with an honest runtime boundary, not a fabricated result.

Optimization-based inverse kinematics for retargeting (libs/aja/optimization-ik/src; package @aja/optimization-ik). It implements a real solver suite under solvers/ — FABRIK, CCD, gradient descent, a Jacobian-transpose/pseudo-inverse solver, and Levenberg-Marquardt — over a shared jacobian.ts (world-transform computation), chain-builder.ts, joint limits/DOF types, and a full-body.ts coordinator. Genuine numeric IK, not a wrapper.

Skeleton mapping and retargeting utilities (libs/aja/skeleton-mapping/src; package @aja/skeleton-mapping). It ships skeleton templates for major formats (templates.ts — Mixamo, MediaPipe, Unity, COCO, SMPL), automatic matching algorithms (matching.ts — name/hierarchy/position/semantic), partial-skeleton handling (partial.ts — upper/lower/hands/face presets), and a topology-agnostic "primal skeleton" converter (converters.ts). The foundational mapping layer the other retargeting libraries and bone-mapping-ui build on.

Tiered motion-asset storage with CDN distribution and automatic archival (libs/aja/asset-storage/src). MotionAssetManager (asset-manager.ts) orchestrates tiered-storage.ts, cdn-distribution.ts, and automatic-archival.ts, with declared STORAGE_TIERS, retention policies, and signed-URL/cache configuration in types.ts. It is the infra-layer (layer:infra) home for pipeline artifacts and their lifecycle transitions.

At-rest and in-transit content security for mocap assets (libs/aja/content-security/src), and a genuinely real crypto implementation: encryption.ts uses Node's node:crypto with aes-256-gcm (real createCipheriv / getAuthTag, AEAD), alongside AES-256-CBC and ChaCha20-Poly1305. It adds envelope encryption / key management (key-management.ts, with LocalKeyProvider + rotation), access control (access-control.ts), transit security (transit-security.ts), and audit logging. This is one of the most fully-real libraries in the area.

A React (.tsx) UI library (libs/aja/avatar-preview-ui/src) for previewing and comparing avatars in the Lilith mocap system. It provides AvatarPreview, ComparisonView (side-by-side variants), ABTestInterface, and QualityOverlay components plus driving hooks (useAvatarPreview, useABTest, useQualityOverlay). It is a layer:ui node — presentation/interaction over the animation and quality data, not a processing stage.

A React UI library (libs/aja/bone-mapping-ui/src) for visually mapping bones/joints between skeleton templates for retargeting. Components include BoneMappingWorkspace, SkeletonViewer, JointMapperPanel, MappingConnection, MotionPreview, and MappingProfileManager, backed by hooks for drag-and-drop, mapping history/undo, profile persistence, and skeleton visualization. It is the human-in-the-loop editor that produces the mappings the skeleton-mapping library consumes.

The avatar/instructor catalog (libs/aja/avatar-library/src) with pre-built instructors, customization, style categories, and a community-marketplace surface. prebuilt/ holds real authored data — named yoga, fitness, and meditation instructors (e.g. MAYA_VINYASA, MARCUS_HIIT) with full skeleton joint hierarchies — exposed through registries and query helpers (getInstructor, searchInstructors, getInstructorsByTier). categories/, customization/, and marketplace/ add the catalog services around that data.

Depth-sensor integration and RGB-D processing (libs/aja/depth-sensing/src). sensor-driver.ts defines an abstract DepthSensorDriver base (state machine, connect()/disconnect() abstract methods) describing RealSense / Azure Kinect / Apple LiDAR — i.e. a driver abstraction, not a bundled native SDK binding. The real processing lives in rgbd-fusion.ts (alignment/fusion), mesh-recovery.ts (body mesh from depth), and depth-quality.ts (assessment / enhancement).

Distributed worker system targeting Kubernetes with autoscaling, GPU node affinity, and fault tolerance (libs/aja/distributed-workers/src). The autoscaler (autoscaler.ts) and fault-tolerance.ts logic are real, but the KubernetesClient in kubernetes.ts simulates the API calls (in-source: "For simulation, we just mark as initialized", "Simulate job creation") rather than binding a live cluster API. Honest framing: a real scheduling/scaling control plane over a simulated K8s client seam.

Cross-domain integration adapters (libs/aja/motion-integration/src) that push motion-pipeline outputs into sibling domains. It carries dedicated adapter + types modules for Yemaya (Creative Studio asset/project/folder mapping), Isis, Bellona, and Sophia, over a shared set of branded IDs (MotionAssetId, MotionJobId, …) and base adapter/auth/retry config in types.ts. This is the outbound boundary layer for the pipeline.

The typed TypeScript SDK (libs/aja/motion-pipeline-sdk/src) for the Lilith Motion Pipeline Service. client.ts exposes a MotionPipelineClient with a fluent job builder, config management, SSE progress streaming, download-URL generation, and webhook management, with typed errors in errors.ts. It is the external entry point consumers use instead of hand-rolling HTTP against the service.

A general pipeline-parallelism engine (libs/aja/pipeline-parallelism/src) for high-throughput inference/data processing. It supports stage overlapping and async stage execution via queues (async-queue.ts, stage.ts), memory pipelining with pooling/GC (memory-pipeline.ts), and multi-GPU stage distribution (multi-gpu.ts), assembled through a createPipeline builder and scheduler.ts. It is the stage-overlap counterpart to batch-inference's batching.

Result aggregation for distributed processing (libs/aja/result-aggregation/src). A result-collector.ts gathers per-chunk worker results, conflict-resolver.ts reconciles overlapping/disagreeing outputs, quality-selector.ts picks the best candidate, and final-assembler.ts stitches the final artifact. It is the gather/merge counterpart to video-chunking's scatter, closing the distributed loop.

Watermarking and provenance for mocap assets (libs/aja/content-watermarking/src). It implements DCT-based spread-spectrum video watermarking (video-watermark.ts, 8×8 blocks with a generated spread-spectrum key — note the mid-frequency DCT coefficient edit is approximated via pixel modification, labelled in-source), LSB motion-data watermarking in joint positions (motion-watermark.ts), a blockchain-like ProvenanceManager (provenance.ts), and a LeakDetectionService (leak-detection.ts). Real spread-spectrum/LSB scheme with a documented coefficient-level approximation.

Intermediate-result caching for motion pipelines (libs/aja/pipeline-cache/src). A pipeline-cache-manager.ts coordinates stage-specific caches — pose-estimation-cache.ts, lifting-cache.ts, retargeting-cache.ts — keyed by a structured PipelineCacheKey, with cache-invalidation.ts and cache-utils.ts handling key derivation and eviction. It lets the expensive stages skip recomputation across pipeline runs.

Human mesh recovery (libs/aja/human-mesh-recovery/src; package @aja/human-mesh-recovery) targeting SMPL/SMPL-X. It implements real SMPL/SMPL-X body math (smplx-body.ts), body fitting/optimization (body-fitting.ts), mesh-to-skeleton conversion (mesh-skeleton.ts), and temporal consistency (temporal-consistency.ts). The HMR2/CLIFF/PyMAF-X neural backbone and person detection in smpl-recovery.ts are explicitly simulated (in-source: "(simulated neural network)", "In production, this would use YOLO …"), with model parameters defaulted rather than loaded — honest seams around a real body-model core.