# Aja Domain — Features > **Aja** — Motion AI and Animation Platform Aja (named after the Yoruba Orisha Aja, guardian of the forest and healer) transforms raw video and motion capture data into production-ready animation. The platform owns the entire pipeline: video ingestion and normalization, AI pose estimation from multiple models, temporal tracking and gap filling, 2D-to-3D pose lifting, full human mesh recovery, motion cleanup and physics-based artifact correction, quality assessment, skeletal retargeting with IK optimization, animation blending, avatar management, and export in every major animation format. Beyond post-production, Aja powers real-time motion capture from webcams and mobile devices, live avatar driving for virtual instruction and VTubing, domain-specific pipelines for fitness form analysis, dance choreography, sports biomechanics, and clinical gait analysis, generative AI motion synthesis from text, and batch processing at scale. Cross-domain integration adapters connect Aja's motion data to Yemaya (film production), Isis (3D asset creation), Bellona (build systems), and Sophia (research and knowledge). Every library in `libs/aja/` corresponds to a specific capability described below. The five application projects are `svc-motion-pipeline`, `svc-motion-ai`, `svc-reference-video`, `cli`, and `docs`. The four domain-specific pipeline modules actually shipped as code are **yoga**, **fitness**, **dance**, and **martial-arts**. The four ML-only pipeline stages (`pose-estimation`, `skeleton-fitting`, `domain-analysis`, `retargeting`) are V2-deferred in the orchestrated pipeline per descope decision `V1-P2-0331`. Sections labelled _(planned)_ have no corresponding code yet. --- ## 1. Video Ingestion and Preprocessing The first stage of the Aja pipeline accepts video from any source and normalizes it for accurate pose estimation. Before pose models can analyze a frame, the video must be stable, consistently lit, and correctly cropped — this stage handles all of that preparation. ### Input Support - **Broad format support**: Aja accepts MP4, MOV, AVI, MKV, WebM, and other common video formats, covering virtually every camera and editing workflow. Variable frame rates and HDR content are handled transparently. - **URL and batch input**: Videos can be submitted by direct file upload, by URL, or as a glob pattern matching many files at once, enabling both interactive and scripted workflows. - **File size tolerance**: Individual files up to 2 GB are supported, accommodating 4K footage or long sessions up to 60 minutes. - **Frame rate range**: 24 FPS through 120 FPS input support ensures compatibility with cinema, broadcast, and high-speed slow-motion cameras. ### Frame Extraction and Quality Enhancement - **Adaptive frame extraction**: Aja can extract every frame, extract at a fixed rate, extract keyframes only, or use motion-based adaptive sampling that concentrates analysis on the highest-motion regions — reducing compute cost on slow segments. - **Video stabilization**: Global motion estimation corrects shaky handheld footage and rolling shutter distortion, producing a smooth camera path that improves pose estimation accuracy. - **Super-resolution and denoising**: Low-resolution or noisy video is enhanced before pose estimation using learned upscaling and denoising models. Legacy interlaced video is deinterlaced automatically. - **Lighting normalization**: Per-frame exposure correction, white balance adjustment, and shadow/highlight recovery ensure consistent illumination across frames — critical for models trained on normalized imagery. - **Person detection and cropping**: An automatic person bounding box detector crops tightly to the subject and dynamically follows them, reducing background distraction for pose models. Multi-person scenes are handled with subject selection tools. - **Video chunking**: Long videos are automatically split into overlapping chunks for parallel processing, then seamlessly stitched back together (`@aja/video-chunking`). --- ## 2. Multi-Model Pose Estimation Aja supports every major open-source pose estimation model and fuses their outputs for the best possible accuracy. Rather than committing to a single model, Aja selects and combines models based on the scene — allowing it to perform better than any individual model in isolation. ### Supported Pose Estimation Models - **MediaPipe Pose**: Google's 33-keypoint world-coordinate model, integrated with hand landmarks (21 points per hand) and face mesh (468 points) for full body coverage. Outputs 3D approximation alongside 2D detections, plus a segmentation mask for background separation. - **MoveNet**: Google's lightweight model in two variants — Lightning (real-time, 17 keypoints) and Thunder (higher accuracy, 17 keypoints). Multi-pose mode supports multiple people in a frame. - **OpenPose**: The pioneering multi-person model with three configurations: Body_25 (25 keypoints including feet), COCO (18 keypoints), and specialized hand (21 per hand) and face (70 points) models. - **ViTPose/MMPose**: Vision Transformer-based pose estimation offering state-of-the-art accuracy. ViTPose-H is the highest-accuracy variant; ViTPose-B trades some accuracy for speed. HRNet backbone variants and the 133-keypoint whole-body model are also available. - **YOLO-Pose**: YOLOv8-Pose provides real-time single-pass detection and pose estimation with persistent person ID tracking across frames. - **AlphaPose**: Multi-person pose with PoseFlow for temporal consistency — particularly strong on crowded scenes and partially occluded figures. ### Model Ensemble and Selection - **Confidence-weighted fusion**: Keypoints from multiple models are fused using per-keypoint confidence weights, so the most confident model wins at each joint. This systematically outperforms any single model. - **Model voting for occlusions**: When one model cannot see a joint due to occlusion, other models' estimates are preferentially used rather than discarding the keypoint entirely. - **Automatic model selection**: Based on scene characteristics (person count, resolution, lighting, available GPU), the pipeline selects the optimal model configuration without user involvement. - **Unified keypoint format**: All models are normalized to a common 68+ keypoint skeleton with standardized coordinate systems and unit scaling for downstream consistency. --- ## 3. Temporal Consistency and Tracking Raw per-frame pose estimates contain frame-to-frame jitter and identity switches. When the same person is estimated independently in each frame, small errors accumulate and the skeleton appears to "jitter" — and if two people swap position, the tracker may assign identities incorrectly. This stage corrects both problems using a combination of tracking, filtering, and learned interpolation. ### Multi-Person Tracking - **DeepSORT**: Combines appearance descriptors with Kalman filtering for robust person re-identification across occlusions. Used when appearance similarity is important. - **ByteTrack**: State-of-the-art tracker that associates even low-confidence detections, reducing identity switches in crowded scenes. - **OC-SORT**: Observation-Centric SORT with improved handling of heavily occluded persons by maintaining track history through long gaps. - **BoT-SORT**: Combines the best elements of ByteTrack and OC-SORT for the most robust multi-person tracking in challenging conditions. - **Persistent person IDs**: IDs are maintained across frames and re-associated after occlusion using re-identification features, ensuring the same skeleton is tracked throughout the clip. ### Temporal Filtering - **One-Euro filter**: Adaptive low-pass filter that removes jitter from fast-moving keypoints while preserving responsiveness during slow, precise movements — particularly important for hand and face tracking. - **Kalman filter**: Predicts keypoint positions during brief occlusions using a motion model, bridging the gap until the keypoint is visible again. - **Savitzky-Golay smoothing**: Polynomial fitting over a sliding window that smooths trajectories while preserving their peaks and valleys — better than naive Gaussian smoothing for motion data. - **Butterworth low-pass filter**: Configurable-cutoff frequency filter for systematic removal of high-frequency noise from any joint trajectory. ### Gap Filling and Interpolation - **Linear interpolation**: Fast gap filling for very short missing spans (1–2 frames). - **Cubic spline interpolation**: Smooth arc-preserving interpolation for multi-frame gaps, maintaining natural motion arcs. - **Physics-based interpolation**: Uses motion dynamics (velocity, acceleration continuity) to generate plausible motion during longer occlusions rather than simple curve fitting. - **Neural interpolation**: Learned motion priors fill gaps in a style consistent with the surrounding motion — essential for long occlusions where physics interpolation diverges. ### Motion Validity Enforcement - **Velocity limit checking**: Detects physically impossible sudden position jumps between frames and treats them as tracking failures to prevent corrupted data downstream. - **Bone length consistency**: Enforces that bone lengths remain constant across frames (they cannot change in a rigid skeleton), correcting estimation drift. - **Joint angle limit validation**: Compares joint angles against anatomical joint limits, flagging anatomically impossible configurations for correction (`@aja/motion-validation`). --- ## 4. 3D Pose Reconstruction (Pose Lifting) Converting 2D pixel-space pose detections into full 3D skeletal data is a core AI capability. This process is called "pose lifting" and it is fundamentally an ill-posed problem — many 3D configurations project to the same 2D image — solved using temporal context and learned human motion priors (`@aja/pose-lifting`). ### Monocular 3D Lifting Models - **VideoPose3D**: Uses temporal dilated convolutions over a window of frames to infer 3D joint positions. Strong baseline with efficient inference. - **PoseFormer**: Transformer-based lifter that attends to long-range temporal context, achieving higher accuracy on complex motions. - **MixSTE**: Mixed spatial-temporal encoder combining spatial attention (across joints) with temporal attention (across frames) for state-of-the-art results. - **MotionBERT**: Pre-trained motion-aware transformer that transfers knowledge from large motion corpora, excelling on novel motion types. - **MHFormer**: Multi-hypothesis architecture that generates multiple candidate 3D poses and selects or fuses the most plausible one, reducing ambiguity errors. ### Depth-Aided Reconstruction - **Depth-conditioned refinement**: Monocular depth estimates from MiDaS or ZoeDepth (which provides metric-scale depth) constrain the pose lifter, reducing the depth ambiguity problem inherent to single-camera capture. - **Camera intrinsics estimation**: Automatic estimation of focal length, principal point, and lens distortion from the video itself, allowing accurate perspective-to-3D projection without a calibration target. - **Scale recovery**: Ground plane estimation and anthropometric constraints recover absolute scale — the absolute position of the person in the real world, not just relative bone orientations. --- ## 5. Human Mesh Recovery Human Mesh Recovery (HMR) goes beyond skeletal pose to reconstruct a dense 3D body surface mesh (`@aja/human-mesh-recovery`). This is used for clothing simulation, detailed body shape analysis, and high-fidelity avatar creation. ### SMPL and SMPL-X Parametric Body Models The SMPL (Skinned Multi-Person Linear) model is a statistical parametric body model learned from thousands of 3D body scans. It represents a human body as a combination of shape parameters (body proportions) and pose parameters (joint rotations), producing a realistic 3D mesh from just ~80 numbers. - **HMR2.0**: The current state-of-the-art single-image HMR model, using a ViT image encoder and transformer decoder to regress SMPL parameters from a single frame. - **CLIFF**: High-accuracy HMR that uses global image context (camera field of view) rather than just the cropped person region, improving absolute position accuracy. - **PyMAF-X**: Mesh-aligned feedback approach that iteratively refines the initial estimate by projecting the mesh back onto the image and correcting misalignment errors. - **4D-Humans**: Video-based HMR that maintains temporal consistency across frames, producing smooth body mesh sequences rather than per-frame jitter. - **WHAM**: World-grounded HMR that estimates motion in the world coordinate frame (not just camera-relative), enabling physics-consistent root motion with gravity. - **SMPL-X full body**: The SMPL-X extension adds 3D hands (MANO model) and expressive face (FLAME model) to the body, providing unified body+hands+face mesh recovery for complete character animation. ### Mesh Utilities - **Mesh-to-skeleton conversion**: Extracts the skeleton hierarchy (joint positions, rotations, bone lengths) from the recovered mesh for downstream retargeting and format export. - **Body shape estimation**: The SMPL beta parameters encode body shape — height, weight distribution, limb proportions — enabling the pipeline to reconstruct a person-specific avatar body that matches the captured individual. --- ## 6. Live Motion Capture — Desktop Capture motion in real time from cameras for avatar driving, interactive applications, and immediate feedback. Desktop live capture is tuned for the lowest possible end-to-end latency so that the avatar responds in sync with the performer. ### Real-Time Processing - **Webcam capture**: Initiate a live capture session from any connected webcam using browser WebRTC or native camera APIs. Virtual cameras (e.g., OBS, Snap Camera) are supported for virtual backgrounds. - **Real-time pose estimation**: GPU-accelerated inference targets sub-50ms end-to-end latency from frame capture to skeleton output, enabling smooth avatar driving at interactive rates. - **Streaming pose lift**: Causal lifting models (those using only past and current frames, not future frames) provide real-time 3D pose lifting during live capture — no post-processing delay. - **GPU-accelerated retargeting**: The entire retargeting pipeline is GPU-accelerated, ensuring that motion from the capture person appears on the target avatar with minimal delay. - **Live avatar driving**: Directly streams the live skeleton to a 3D renderer (browser, Unity, Unreal) for real-time avatar animation — used for virtual instruction, VTubing, and interactive experiences. ### Game Engine Integration Aja supports five streaming protocols, allowing live motion data to be delivered to any runtime environment: - **Unity/Unreal Live Link**: Native integration protocols for delivering live motion data directly into game engine scene graphs — no middleware or intermediate conversion required. - **WebRTC data channel**: Low-latency browser-native streaming, ideal for web-based avatar experiences. - **WebSocket with binary frames**: Efficient binary skeleton streaming over persistent WebSocket connections. - **gRPC streaming**: High-performance bidirectional streaming for server-to-server and native application integration. - **Custom UDP protocol**: Minimal-overhead UDP streaming for the lowest possible latency in local network applications. ### Edge and Reliability - **Edge deployment**: Edge GPU inference nodes are selected by geographic proximity to minimize network latency, with automatic fallback to cloud if edge is unavailable. - **Latency monitoring**: End-to-end and per-stage latency is continuously measured, logged, and alerted on threshold violations. --- ## 7. Live Motion Capture — Mobile Capture motion on iOS and Android devices using on-device ML for privacy and low latency. Mobile capture is implemented in the Motion AI service's `mobile-live-capture` module (`apps/aja/svc-motion-ai/src/mobile-live-capture/`: `ios-capture.ts`, `android-capture.ts`, `on-device-processing.ts`, `mobile-streaming.ts`). ### iOS Capture - **ARKit body tracking**: Native Apple ARKit body tracking provides robust 3D joint estimation using the device's depth-enhanced camera system on iPhone and iPad Pro. - **Core ML pose models**: On-device inference using Apple's Neural Engine for fast, private pose estimation without cloud round-trips. - **LiDAR depth integration**: iPhone/iPad Pro LiDAR scanners provide metric depth data that dramatically improves 3D pose accuracy compared to monocular estimation. - **Live streaming to server**: When on-device processing is insufficient, captured video streams to the Aja server with adaptive bitrate adjustment for network quality. ### Android Capture - **ML Kit pose detection**: Google's on-device ML Kit provides full-body pose detection directly on Android devices. - **TensorFlow Lite models**: Optimized TFLite models run on Android NPU/GPU for sub-50ms inference. - **ARCore depth integration**: ARCore's depth API provides depth maps on supported Android devices for enhanced 3D accuracy. - **Hybrid on-device/cloud processing**: The system selects the optimal split between on-device and cloud processing based on device capability and network quality. ### Mobile Infrastructure - **Adaptive bitrate streaming**: Video quality adjusts dynamically to available bandwidth, maintaining continuity over variable mobile connections. - **Offline buffering and resume**: Captures are buffered locally and transmitted when connectivity is restored, preventing data loss on unstable connections. - **Battery-efficient processing**: Mobile GPU acceleration and lightweight model variants reduce battery drain for extended capture sessions. --- ## 8. Depth Sensing Dedicated library for depth camera integration and RGB-D data fusion (`@aja/depth-sensing`). Depth cameras eliminate the depth ambiguity problem of monocular video by directly measuring the distance of each pixel from the camera, enabling more accurate 3D joint positions without the uncertainty of monocular pose lifting. - **Intel RealSense D4xx**: Full support for Intel RealSense depth cameras (D435, D455, D457) with SDK integration for structured light depth maps. - **Microsoft Azure Kinect**: Integration with the Azure Kinect DK's time-of-flight depth sensor, which provides high-quality depth at 120 fps. - **Apple LiDAR (iPhone/iPad Pro)**: Reads LiDAR depth frames from Apple devices, providing dense metric-scale depth maps at 30 fps. - **Stereolabs ZED cameras**: Support for the ZED 2i and ZED X stereo cameras, which provide both depth and RGB at high resolution. - **Orbbec cameras**: Support for Orbbec's structured light depth cameras. - **RGB-D fusion**: Depth data and 2D pose detections are fused — depth guides 3D keypoint extraction directly (no lifting needed) while RGB provides appearance context for identity and confidence. - **Depth hole filling**: Missing depth values (common near object boundaries) are filled using spatial and temporal interpolation. - **Body-only depth filtering**: Clothing segmentation masks filter depth to the body region only, preventing background depth from corrupting body reconstruction. --- ## 9. Multi-View Reconstruction Reconstruct high-accuracy 3D motion from multiple synchronized camera views (`@aja/multi-view-reconstruction`). With two or more cameras at different angles, 3D positions can be triangulated geometrically rather than estimated — eliminating depth ambiguity entirely and significantly improving accuracy on complex movements where a single camera is occluded. ### Camera Calibration Before triangulation can work, each camera's internal parameters and its position relative to the others must be known precisely: - **Checkerboard and ChArUco calibration**: Automatic intrinsic calibration (focal length, distortion) of each camera from calibration target images placed in the capture space. - **Extrinsic calibration**: Relative positions and orientations of all cameras are determined automatically, with bundle adjustment refinement for maximum geometric accuracy. ### 3D Reconstruction Once cameras are calibrated, 3D positions are recovered from matching 2D observations across views: - **Direct Linear Transform (DLT) triangulation**: The simplest and fastest triangulation method — given corresponding 2D points across cameras, DLT recovers 3D position algebraically. - **RANSAC-based robust triangulation**: Outlier-resistant triangulation that identifies and down-weights mismatched keypoints before computing the 3D position. - **Confidence-weighted averaging**: Views with higher keypoint confidence contribute more to the fused 3D estimate. - **Gaussian Process fusion**: Multi-view estimates are fused in a probabilistic framework that accounts for per-view uncertainty for a principled statistical estimate. ### Synchronization Multi-camera rigs require frame-accurate time alignment before triangulation is valid: - **Audio/visual synchronization**: Multi-camera clips are synchronized using audio cross-correlation (e.g., a clap), visual flash events, or embedded timecodes, down to sub-frame precision. - **Volumetric capture integration**: Integration with professional volumetric capture setups including Intel RealSense multi-camera arrays, Azure Kinect multi-camera, and Microsoft Mixed Reality Capture Studios. --- ## 10. Motion Format Conversion Import and export across 13+ animation formats, with full round-trip fidelity (`@aja/motion-formats`). A central internal representation (`AnimationClip`) means every other library works with the same data model, regardless of which format the user submitted or requested. ### Supported Formats The table below lists every format, whether it can be read, written, and a brief description of its use case. | Format | Read | Write | Description | | ------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- | | BVH (Biovision Hierarchy) | Yes | Yes | Universal MoCap exchange standard — the most widely used format for skeleton animation transfer between applications | | FBX (Autodesk) | Yes | Yes | Game engine and DCC tool integration (Unity, Unreal, Maya, Blender) with embedded animation curves | | glTF/GLB | Yes | Yes | Web-ready 3D format with animation support — single-file GLB for easy distribution | | USD/USDA (Pixar) | Yes | Yes | USD with UsdSkel schema for VFX pipelines and Omniverse workflows | | Alembic (ABC) | Yes | Yes | VFX industry standard for baked geometry caches and skeletal animation | | C3D | Yes | No | Biomechanics optical marker standard from Vicon, OptiTrack, and other lab-grade motion capture systems | | TRC | Yes | No | Marker trajectory files from lab biomechanics systems | | ASF/AMC (Acclaim) | Yes | Yes | Classic motion capture exchange format from early academic MoCap research datasets | | Collada (DAE) | Yes | Yes | Cross-platform 3D exchange used by Blender and many authoring tools | | MDD | Yes | Yes | Point cache displacement format used in VFX | | PC2 (Point Cache v2) | Yes | Yes | 3ds Max point cache format | | OBJ sequences | Yes | No | Animated mesh sequences from volume capture or simulation | | Custom JSON | Yes | Yes | Oshun's internal motion format with full metadata and provenance | ### Conversion Capabilities - **Coordinate system remapping**: Handles Y-up vs. Z-up axis conventions and left- vs. right-handed coordinate systems between formats without manual remapping by the user. - **Frame rate resampling**: Converts animation from any source frame rate to any target frame rate using motion-aware resampling rather than naive linear interpolation. - **Scale normalization**: Automatically converts between centimeter and meter units (a common mismatch between formats) while preserving motion scale. - **Clip database**: Every imported or exported motion clip is stored in a searchable database with metadata, tags, and versioning. Clips can be searched, categorized, and retrieved by multiple criteria. - **Multi-take support**: FBX and ASF/AMC files can contain multiple animation takes; Aja imports all takes and allows exporting selected takes. --- ## 11. Motion Processing and Cleanup Transform raw captured motion into production-quality animation (`@aja/motion-processing`). Even after temporal tracking, captured motion contains noise, artifacts, and physically impossible configurations that would look wrong when played back on a character. This library corrects them. ### Noise Reduction and Smoothing - **Frequency-domain filtering**: Decomposes joint trajectories using FFT and suppresses components above a configurable frequency threshold — removes high-frequency noise while preserving intended motion. - **Wavelet denoising**: Multi-resolution wavelet decomposition that removes noise at specific frequency bands while preserving motion discontinuities (like impact frames) that Fourier methods would blur. - **Motion-aware adaptive filtering**: Per-joint noise profiling applies stronger filtering to noisy joints and lighter filtering to clean ones — no single global filter parameter for all joints. - **Per-joint noise profiling**: Each joint in each clip has its own noise characterization, enabling the filter to be customized per joint rather than applying a one-size-fits-all approach. ### Artifact Correction - **Foot sliding correction**: Ground contact detection identifies frames where feet should be planted on the ground; IK-based foot locking constrains foot positions during contact phases, eliminating the "skating" artifact common in video-derived animation. The blend in/out of contact is smooth to avoid pops. - **Physics-based cleanup**: Center-of-mass trajectory smoothing, angular momentum conservation, balance validation, and joint torque limit enforcement remove physically implausible configurations from the animation. - **Root motion extraction**: Separates the global translation and rotation of the root bone from the in-place animation, producing both a root motion track (for engine locomotion systems) and an in-place loop — both are necessary for game engine integration. ### Motion Editing - **Motion retiming and time warping**: Adjust the tempo of a motion clip — slow down, speed up, or apply a non-linear time warp curve — without changing the spatial motion. - **Beat synchronization**: Align motion timing to music beats for dance and choreography applications. - **Gap filling**: Short gaps use spline interpolation; long gaps use learned motion synthesis that produces style-consistent motion rather than mechanical interpolation. - **Motion segmentation**: Automatically detects action boundaries within a long clip to identify natural cut points, loop-ready segments, and transition zones. --- ## 12. Motion Quality Assessment Automated multi-dimensional quality scoring ensures only production-quality motion is delivered (`@aja/motion-quality`). Quality scoring runs as a dedicated pipeline stage, allowing operators to configure minimum thresholds and automatically reject or flag poor-quality inputs before they reach retargeting or delivery. ### Technical Quality Metrics These metrics measure objective properties of the animation data that correlate with visible artifacts: - **Jitter score**: Measures high-frequency noise (acceleration variance) in joint trajectories — the primary indicator of tracking noise contamination. Lower is better; zero is ideal. - **Foot sliding distance**: Quantifies the total sliding distance of foot contacts against the ground plane during contact phases. Even small values are visible to the eye in real-time animation. - **Bone length variance**: Measures how much bone lengths deviate across frames — perfect tracking produces zero variance; this reveals estimation drift over the course of a clip. - **Joint angle smoothness**: Second-derivative (jerk) analysis of joint angle trajectories, revealing unnatural snapping or jerking motions that would look wrong on screen. - **Physical plausibility score**: Composite score combining joint limit violations, self-collision detection, center-of-mass stability, and angular momentum conservation. ### Accuracy Metrics (When Ground Truth Is Available) When a ground-truth reference recording exists (from a lab-grade motion capture system), these standard benchmark metrics quantify estimation accuracy: - **MPJPE (Mean Per Joint Position Error)**: The standard benchmark metric — average Euclidean distance between predicted and ground truth joint positions in 3D space, measured in millimeters. Used for academic comparisons. - **PA-MPJPE (Procrustes-Aligned MPJPE)**: Removes global alignment differences (scale, rotation, translation) before computing MPJPE, measuring pure pose shape accuracy independent of root position. - **PCK (Percentage of Correct Keypoints)**: Percentage of keypoints within a distance threshold of ground truth, useful for comparing methods at different accuracy levels. - **AUC (Area Under Curve)**: Area under the PCK curve across a range of distance thresholds — a single-number summary of pose estimation accuracy across all scales. ### Perceptual Quality Metrics Technical metrics do not always capture what looks natural to a human observer. These learned metrics fill that gap: - **Motion naturalness score**: A learned perceptual quality score that predicts how natural or unnatural the motion looks to a human observer — capturing qualities that technical metrics miss. - **Style consistency score**: Measures how consistently a particular movement style is maintained throughout the clip, detecting stylistic drift. - **Visual quality rating**: Composite rating combining temporal coherence, naturalness, and smoothness for human-readable quality communication. ### Quality Management - **Per-frame quality annotation**: Frame-level quality scores identify the worst-quality regions in a clip, guiding targeted cleanup rather than blanket re-processing. - **Automated QA pipeline**: Clips below configurable quality thresholds are automatically rejected; quality reports are generated; borderline cases are flagged for manual review. --- ## 13. Motion Retargeting and Skeleton Mapping Transfer motion from any source skeleton to any target skeleton, preserving intent and compensating for proportional differences. "Retargeting" is the process of taking animation from one character rig and making it play correctly on a different character with different proportions and joint naming (`@aja/skeleton-mapping`). ### Skeleton Mapping - **Pre-built skeleton template library**: Templates for Mixamo, MediaPipe, Unity Humanoid, COCO, H36M, OpenPose, SMPL, and fitness-specific skeleton conventions — covering every major game engine, web, and research pipeline. - **AI-powered automatic bone matching**: A multi-strategy matcher combines bone name pattern matching (with fuzzy matching and alias resolution), hierarchical structure matching (tree depth, sibling/child relationships), proportional similarity matching, and semantic joint type matching (understanding that "hip" and "pelvis" are equivalent). This handles even novel skeleton configurations without manual mapping. - **Interactive bone mapping UI**: A visual drag-and-drop interface renders both skeletons as SVG overlays and lets users manually connect bones, preview the retargeted result, scrub through the timeline, and save/load mapping profiles (`@aja/bone-mapping-ui`). - **Partial skeleton retargeting**: Supports retargeting only the upper body, lower body, hands, or face independently — useful for blending captured upper-body motion onto a procedurally animated lower body. ### Proportional Adaptation (`@aja/proportional-adaptation`) When source and target characters have different body proportions (e.g., retargeting an adult's motion to a child avatar), naive rotation transfer causes foot skating, hand penetration, and misaligned contacts. - **Limb length scaling**: Adapts reach distance so contacts (floor, wall, prop) are correctly maintained on the target despite different limb lengths. - **Reach adjustment**: End effector positions are reprojected through the target skeleton's IK chain to compensate for arm and leg length differences. - **Ground contact adaptation**: Foot contact frames are detected on the source and reconstructed on the target at the correct ground plane height for the target's limb lengths. - **Collision avoidance**: Self-collision between target skeleton segments (e.g., hands passing through the torso) is detected and resolved automatically. --- ## 14. Inverse Kinematics and Pose Optimization Optimization-based retargeting for precise end-effector control (`@aja/optimization-ik`). Inverse kinematics (IK) is the mathematical problem of finding joint angles that position an end effector (hand, foot, head) at a desired position in space. IK is used both to clean up retargeted poses and to enforce contact constraints (feet on ground, hands on surfaces). ### IK Solvers Three solvers cover different trade-offs between speed, accuracy, and joint constraint support: - **Jacobian IK**: Uses the linearized relationship between joint velocities and end-effector velocities (the Jacobian matrix) to iteratively solve for joint angles. Handles redundant chains well. - **CCD (Cyclic Coordinate Descent)**: Simple, fast iterative solver that adjusts one joint at a time from the leaf to the root. Converges quickly for typical limb chains. - **FABRIK (Forward And Backward Reaching Inverse Kinematics)**: Position-based solver that alternates forward and backward passes along the bone chain. Excellent performance on multi-segment chains with joint limits. ### Pose Optimization Beyond basic IK, the optimizer can enforce multiple simultaneous constraints using gradient-descent methods: - **Joint limit constraints**: Hard angle limits prevent anatomically impossible configurations. - **Position and orientation targets**: End effectors can be constrained to world-space positions and orientations simultaneously. - **Pole vector constraints**: Specify a "pole vector" direction to control elbow/knee bend direction during IK solving — prevents knee/elbow flipping artifacts that look unnatural. - **Ground contact constraints**: Feet are locked to the ground plane during detected contact phases, eliminating foot sliding. - **Center-of-mass constraints**: Constrain the skeleton's center of mass to stay within the support polygon for physically plausible balance. - **Trajectory optimization**: Gradient descent, Adam optimizer, momentum, and Levenberg-Marquardt methods are available for trajectory-level optimization of full-body poses over time. --- ## 15. Neural and Semantic Retargeting Higher-quality retargeting through learned representations (`@aja/neural-retargeting`, `@aja/semantic-retargeting`). Geometric retargeting works well when source and target skeletons are similar, but fails on large morphological differences or when the literal joint mapping would destroy the intent of the motion. Neural and semantic methods address both problems. ### Neural Retargeting - **SAN (Skeleton-Aware Network)**: A neural network that learns to retarget motion between skeleton pairs by encoding the source motion in a skeleton-independent latent space and decoding it for the target skeleton's specific structure — enabling retargeting without explicit bone mapping. - **Cross-structure motion transfer**: Handles extreme morphological differences between source and target — including bipedal-to-quadrupedal or human-to-creature retargeting — using learned correspondence rather than geometric bone mapping. - **Style-preserving retargeting**: The neural model explicitly separates motion content (what movement is being performed) from motion style (how it is performed), allowing content transfer while preserving the source's movement style. - **Unpaired skeleton transfer**: Training pairs of identical motion on source and target are not required — the model learns from unpaired motion corpora, making it practical for novel skeleton targets. ### Semantic Retargeting - **Intent preservation**: Rather than directly mapping joint rotations, semantic retargeting understands the intent of the motion (e.g., "reach for the door handle") and reconstructs that intent on the target skeleton, even if the joint-level mapping would fail (`@aja/semantic-retargeting`). - **Action-specific retargeting rules**: Different retargeting strategies are applied based on the action type — a punch retargets differently from a yoga pose because they have different biomechanical priorities. - **Context-aware joint priorities**: Joints that carry the primary semantic meaning of a motion (e.g., the striking hand during a punch) are prioritized over secondary joints during retargeting. --- ## 16. Blend Shape and Facial Retargeting Transfer facial expressions and secondary motion to target avatars (`@aja/blend-shape-retargeting`). Blend shapes (also called morph targets) are pre-sculpted mesh deformations corresponding to facial muscle actions, controlled by a weight value from 0 to 1. Facial retargeting maps the captured expression weights from the source face to the target avatar's blend shape rig, which may use different shape counts and naming conventions. - **Blend shape extraction**: Extracts facial blend shape weights from video using a face model — each blend shape corresponds to a specific facial muscle action (e.g., mouth corner raiser, brow lowerer). - **Facial expression transfer**: Extracted expression parameters are mapped to the target avatar's face rig, which may use different blend shape names or different numbers of shapes. The mapping resolves these differences automatically. - **Eye tracking and gaze transfer**: Pupil and iris tracking transfers gaze direction to the target avatar, critical for realistic eye contact during virtual instruction or performance. - **Micro-expression preservation**: Fine-grained expression nuances are preserved during transfer rather than being filtered out as noise — essential for naturalistic avatar performance. - **Secondary motion transfer**: Muscle deformation parameters, soft tissue jiggle parameters, and cloth simulation interaction parameters are transferred alongside the primary skeleton motion. --- ## 17. Animation Blending Blend between multiple motion clips for seamless transitions and layered animation (`@aja/animation-blending`). Blending is used both in post-processing (e.g., smoothly combining a captured walk with an idle) and in real-time applications (e.g., crossfading between animations in a game engine). - **Two-clip cross-fading**: Blend between two animation clips with configurable blend duration and easing curves, producing smooth transitions without pops or discontinuities. - **Layer-based blending**: Blend full-body animation with upper-body or lower-body overrides independently — for example, apply a wave gesture to the upper body while the lower body continues a walk cycle. - **Additive animation layers**: Stack additive animations on top of base animations — for example, add a breathing motion on top of all other animations without disrupting the base animation. - **Blend extracted motion with idle**: Smoothly blend between a live-captured or retargeted clip and the character's idle animation, controlled by an incoming activity threshold. - **Clip transition optimization**: Automatic detection of the best transition point between clips (e.g., at a matching pose) minimizes visible pops during manual clip-to-clip switching. --- ## 18. Skeleton and Avatar Management Manage skeleton definitions and bind avatars for the full pipeline (`@aja/avatar-library`, `@aja/avatar-integration`, `@aja/avatar-preview-ui`). Avatar management bridges the gap between the abstract skeleton data produced by the pipeline and the concrete 3D characters that end users see. ### Skeleton Library - **Hierarchical skeleton definitions**: Store complete skeleton hierarchy definitions — bone names, parent-child relationships, bind poses, and joint orientation conventions — for any number of target skeletons. - **Custom skeleton registration**: Register novel skeleton configurations with the `registerSkeletonTemplate` API, making them available for automatic matching and retargeting. - **Skeleton normalization**: Normalize bone orientations and naming conventions to a canonical form, simplifying cross-skeleton operations. ### Avatar Binding and Preview - **Avatar binding**: Bind a motion clip or live stream to a specific avatar definition, associating the skeleton with a 3D mesh, material, and rig for preview and export. - **Pre-configured avatar presets**: Out-of-the-box bindings for Mixamo characters, Unreal Engine Mannequins, and MetaHuman rigs — the most common game development targets. - **Real-time 3D preview**: An interactive 3D viewport renders the retargeted motion on the bound avatar in real time with playback controls and timeline scrubbing. - **Side-by-side comparison**: Source motion and retargeted motion can be viewed simultaneously in a split-screen layout for quality assessment. - **Avatar customization and marketplace**: Pre-built instructor avatars in multiple styles (realistic, stylized, generic coach), with a community avatar marketplace for custom character rigs. --- ## 19. Domain-Specific Pipelines Pre-configured motion processing pipelines optimized for specific application domains (`@aja/domain-motion-pipelines`). Rather than requiring each application to tune pipeline parameters manually, Aja ships pipelines that are pre-tuned for the characteristics of specific types of motion. `@aja/domain-motion-pipelines` ships four pipeline modules: **yoga**, **fitness**, **dance**, and **martial-arts** (`yoga-pipeline.ts`, `fitness-pipeline.ts`, `dance-pipeline.ts`, `martial-arts-pipeline.ts`). The motion-pipeline service's `PipelineDomain` enum is correspondingly `yoga | fitness | dance | martial-arts | general`. The Sports, Medical, Gaming, and Film/VFX pipelines described below are capability targets, not yet shipped as their own pipeline modules in the library. ### Fitness Pipeline (`@aja/fitness-animation`) The fitness pipeline captures and analyzes exercise motion with clinical-grade precision, enabling AI coaching and form analysis. - **Exercise form analysis**: Compares joint angles and body segment orientations frame-by-frame against ideal form templates for each exercise, generating per-joint accuracy scores and flagging deviations. - **Repetition counting**: Detects the cyclic pattern of an exercise movement (e.g., squat depth) and counts completed repetitions from continuous capture. - **Range of motion measurement**: Quantifies the full angular excursion of each joint throughout an exercise — diagnostically valuable for rehabilitation and mobility assessments. - **Pace and tempo tracking**: Measures exercise cadence (reps per minute), eccentric/concentric split, and consistency of tempo across sets. ### Dance Pipeline - **High-accuracy choreography capture**: Tuned for fast, expressive, large-amplitude dance motion — higher filtering thresholds, motion-aware sampling, and temporal consistency parameters optimized for dance dynamics. - **Style transfer**: Apply the captured movement style of one dancer to another performer's motion, preserving individual technique while sharing choreography. - **Beat synchronization**: Align motion timing to music beats using onset detection and time-warping, ensuring choreography is always musically synchronized. ### Martial Arts Pipeline The martial-arts pipeline analyzes combat motion through dedicated services for style detection, strike detection, stance analysis, footwork analysis, and combat-sequence tracking. - **Style detection**: A `MartialArtsStyleDetectionService` matches captured motion against a style-signature database (`MARTIAL_ARTS_STYLE_DATABASE`). - **Strike and stance analysis**: A `StrikeDetectionService` recognizes strikes against `STRIKE_SIGNATURES`; a `StanceAnalysisService` and `FootworkAnalysisService` evaluate stance and footwork quality. - **Combat sequence tracking**: A `CombatSequenceTracker` follows multi-strike combinations across a continuous capture. ### Sports Pipeline - **Athletic technique analysis**: Sport-specific form analysis (e.g., golf swing, baseball pitch, tennis serve) compares captured motion against biomechanical models of optimal technique. - **Biomechanical breakdown**: Compute joint-level angular velocities, estimated forces, and power outputs for coaching and performance analysis. - **Comparison to reference**: Side-by-side or overlay comparison of an athlete's motion to a reference model (e.g., elite athlete template). ### Medical Pipeline - **Clinical gait analysis**: Extracts spatial and temporal gait parameters — cadence, stride length, step width, foot clearance, double support time — in formats compatible with clinical reporting. - **Rehabilitation tracking**: Tracks movement quality metrics over multiple therapy sessions, quantifying recovery progress and detecting regressions. - **Clinical assessment reports**: Generates structured movement assessment reports with reference range comparisons suitable for clinical review. ### Gaming Pipeline - **Real-time gesture recognition**: Recognizes a library of discrete gestures in real time for game control input — without the need for dedicated motion controllers. - **Loop-ready animation export**: Post-processes captured clips to produce seamless looping animations directly usable in game engine state machines. ### Film/VFX Pipeline - **High-fidelity capture mode**: Maximum-accuracy mode that trades speed for precision — uses all available models with ensemble fusion, slower physics cleanup, and highest-quality interpolation. - **Integrated facial performance capture**: Simultaneous body and facial expression capture in a single pipeline, delivering fully animated character performance to VFX. - **Clean VFX-ready export**: Delivers Alembic and FBX output with clean data (no jitter, no foot slide, correct coordinate systems) ready for VFX pipeline ingestion. ### Film Deliverable Packaging (`@aja/film-pipeline`) A dedicated library packages an `AnimationClip` into a film/VFX delivery package with a validated manifest. This is distinct from the Film/VFX pipeline above — it handles packaging and validation of the final deliverable, not capture-time processing. - **Delivery tiers and deliverables**: A `FilmPipelineInput` (production, shot, take, actor IDs; frame rate of 24/25/30/48/50/60; SMPTE timecode start; color pipeline; coordinate system; unit scale) is built into a `FilmPipelineOutputPackage` for a delivery tier of `editorial`, `vfx-review`, `final-vfx`, or `archive`. Each requested deliverable targets one of FBX, USD/USDA/USDC/USDZ, Alembic, BVH, or GLB. - **Validation gating**: The packager produces `FilmPipelineValidationIssue` entries — missing frames, frame-rate mismatch, invalid or missing timecode, a missing required deliverable, missing metadata/skeleton/blend-shapes, or an unsupported final-delivery format — each `warning` or `blocking`. The package `status` is `ready`, `needs-attention`, or `blocked`, with a `deliveryScore` and a versioned `manifestJson` (`aja-film-pipeline/v1`). --- ## 20. AI Motion Synthesis and Enhancement Machine learning capabilities that go beyond motion capture to generate and enhance motion. These are implemented in the Motion AI service's `motion-enhancement` module (`apps/aja/svc-motion-ai/src/motion-enhancement/`), which defines motion synthesis, style transfer, super-resolution, and physics-refinement code. Separately, `@aja/model-optimization` handles model compression and quantization (ONNX optimization, TensorRT compilation, operator fusion, mixed precision) for the inference models that power these capabilities. ### Motion Diffusion Models Diffusion models are generative AI systems that learn the distribution of natural human motion and can sample new motion from that distribution — conditioned on text descriptions, style prompts, or partial motion constraints. - **MDM (Motion Diffusion Model)**: A denoising diffusion probabilistic model for unconditional and text-conditioned human motion generation. - **MLD (Motion Latent Diffusion)**: Operates in a compact latent space rather than raw joint trajectories, enabling faster sampling and richer conditioning. - **MotionGPT**: A language model treating motion sequences as "tokens," enabling natural language prompts like "a person walks then jumps over an obstacle." - **Motion in-betweening**: Given keyframe poses at specific times, AI fills the motion between keyframes in a style-consistent manner. Control point-based generation allows authoring complex motion by placing pose targets at key moments. ### Motion Style Transfer - **Content/style disentanglement**: Neural networks separate what a motion does (content) from how it is done (style). A "confident walk" and a "shy walk" share content but have different styles. - **Unpaired style transfer**: Style can be transferred from reference clips without requiring paired training data — the model learns style as a global motion feature. - **Style blending**: Interpolate between multiple styles (e.g., 50% energetic + 50% graceful) for fine-grained stylistic control. - **Style categories**: Pre-defined style library covering age-based variation (young, adult, elderly), energy levels (calm, energetic), personality styles (confident, shy), and activity-specific styles (yoga, dance, combat). ### Motion Super-Resolution - **Temporal super-resolution**: Upsample animation from 30 fps to 60 fps or 120 fps using motion-aware interpolation that generates physically plausible in-between frames. - **Spatial super-resolution**: Recover fine detail — finger motion, micro-expressions, weight shifts — that low-resolution or long-range capture misses. - **Secondary motion synthesis**: Add organic secondary motion not captured: breathing, soft tissue oscillation, weight shift details, and natural micro-jitter that makes animation feel alive. ### Physics-Based Refinement - **Rigid body dynamics layer**: A physics simulation pass that replaces physically implausible poses with dynamically consistent ones — maintaining momentum and contact dynamics. - **Physics-informed neural networks**: Neural networks with physics equations embedded in their loss functions, ensuring outputs respect physical laws even in novel motion regimes. - **Contact dynamics learning**: Learned models for ground contact, hand grasp, and body-object interactions produce realistic contact behaviour without requiring explicit physics simulation. --- ## 21. Batch Processing and Scale Process large volumes of motion data efficiently across distributed infrastructure. The batch and scaling infrastructure allows Aja to process thousands of video files by distributing work across GPU worker pools and parallelizing within individual jobs. ### Batch Infrastructure - **Batch inference**: Submit thousands of video files as a batch with priority ordering and deadline scheduling. The system distributes work across available workers automatically (`@aja/batch-inference`). - **Video chunking**: Long videos are automatically split into overlapping chunks for parallel processing. Chunk results are seamlessly stitched with cross-chunk motion continuity (`@aja/video-chunking`). - **Pipeline parallelism**: Adjacent pipeline stages run in parallel using a pipelined streaming architecture — stage N+1 begins processing the output of stage N before stage N finishes the full input (`@aja/pipeline-parallelism`). - **Distributed workers**: A worker pool coordinator manages horizontal scaling across multiple GPU workers, with load balancing and health monitoring (`@aja/distributed-workers`). - **Result aggregation**: Results from distributed workers are collected, validated, and merged into unified per-clip outputs, including quality metrics aggregated across chunks (`@aja/result-aggregation`). ### Caching and Efficiency - **Pipeline caching**: LRU (Least Recently Used) cache stores expensive intermediate results (e.g., pose estimation features) so that repeated processing of similar inputs reuses cached results rather than recomputing (`@aja/pipeline-cache`). - **Model warm-up and caching**: Inference models are loaded and kept warm in GPU memory between jobs, eliminating cold-start latency for batch processing. - **Adaptive quality scaling**: For batch workloads where speed matters more than maximum accuracy, the pipeline automatically selects lighter models and relaxed quality thresholds. ### Asset Storage and Delivery (`@aja/asset-storage`) Processed motion assets are stored and delivered through a tiered storage system with CDN distribution — ensuring fast retrieval for recently accessed assets and cost-efficient archival for older ones. - **Tiered storage management**: Motion assets automatically migrate between storage tiers (hot/warm/cold/archive) based on access frequency and age — keeping frequently accessed clips in fast storage while archiving stale ones to lower-cost object storage. - **CDN distribution**: Finished motion exports (BVH, FBX, glTF) are published to a CDN for low-latency download by geographically distributed consumers — game studios, VFX pipelines, and web clients all receive files from the nearest edge node. - **Signed URL delivery**: Time-limited signed URLs provide secure access to output files without exposing raw storage credentials — each download link expires after a configurable duration. - **Asset retention policies**: Configurable rules per customer or project tier determine how long raw intermediate assets (pose estimates, lifted 3D poses) are retained before automatic deletion, separate from the policies governing final output files. - **Archive status tracking**: Track archival status per asset — whether it is retrievable immediately, being retrieved from archive, or has been permanently deleted per retention policy. --- ## 22. Pipeline Configuration and Job Management Full control over pipeline stages, job lifecycle, and result delivery. ### Configuration - **Stage enabling/disabling**: Every pipeline stage can be individually enabled or disabled, allowing a custom pipeline that skips unnecessary steps (e.g., skip cleanup for exploratory analysis). - **Quality vs. speed presets**: Pre-configured profiles ranging from "draft" (fastest, lowest quality) to "master" (slowest, maximum quality) with intermediate options. - **Per-stage parameters**: Every stage exposes its tunable parameters (filter cutoffs, confidence thresholds, cleanup aggressiveness) for expert configuration. ### Job Lifecycle - **Job submission**: Single files or batch lists can be submitted via API, CLI, or web UI with priority levels and optional deadline. - **Real-time progress**: Job progress is streamed via Server-Sent Events (`GET /v1/pipeline/jobs/:jobId/stream`) — stage-by-stage completion percentage and progress messages — and can also be polled. - **Job cancellation**: In-progress jobs can be cancelled; partially completed stages are committed so the job can be resumed from the last checkpoint. - **Automatic retry**: Transient failures (GPU OOM, network timeouts) trigger automatic retry with exponential backoff and jitter up to a configurable maximum. - **Result webhooks**: Webhook callbacks deliver job completion notifications (including download URLs and quality metrics) to caller-specified endpoints. --- ## 23. Reference Video Management A searchable library of reference motion videos for comparison, benchmarking, and instructional content. Reference videos allow teams to share annotated examples, compare against ideal form, and build training datasets without reprocessing the same sources repeatedly. - **Multi-source ingestion**: Videos can be uploaded by drag-and-drop, imported by URL, downloaded from YouTube (legal content only), or submitted as batch uploads. - **Automatic metadata extraction**: Scene detection, person count, activity classification, duration, resolution, and frame rate are extracted automatically from each ingested video. - **Semantic search**: Find reference videos by natural language description (e.g., "someone doing a yoga backbend") or by motion similarity — upload an example and find similar motions in the library. - **Annotation tools**: Annotate reference videos with pose markers, timing notes, and written guidance visible to other team members. - **Collaborative sharing**: Share annotated reference videos with team members with access control, and track who has viewed and used each reference. - **Version management**: Multiple versions of reference materials can be tracked — original, corrected, and annotated versions are kept separately. --- ## 24. Privacy and Security Protecting personal biometric data captured during motion processing. Because motion capture records the movements of real people, biometric data must be treated with care — Aja builds privacy and security into the pipeline itself rather than relying on application-layer safeguards. - **Privacy-preserving processing**: Raw video frames are deleted immediately after motion extraction — the biometric pose data is retained but the identifiable video imagery is not, satisfying privacy requirements for video of real people (`@aja/privacy-protection`). - **Consent management**: Every video subject's data consent is tracked with a consent record, consent version, expiration date, and purpose limitation. Consent can be revoked, triggering immediate deletion of associated data (`@aja/consent-management`). - **Content security**: AES-256 encryption for data at rest and TLS for data in transit. Role-based access control prevents unauthorized access to motion data (`@aja/content-security`). - **Digital watermarking**: Ownership watermarks are embedded invisibly in exported animation files (BVH, FBX) so that unauthorized redistribution can be traced back to the source (`@aja/content-watermarking`). - **Content moderation**: Source videos are screened for inappropriate content before processing is allowed to proceed (`@aja/content-moderation`). - **Data retention policies**: Configurable retention rules determine how long motion data, source video (if retained), and job metadata are kept before automatic deletion (`@aja/data-retention`). --- ## 25. SDKs and Developer Tools Programmatic access to the full Aja pipeline for developers and researchers. ### TypeScript SDK (`@aja/motion-pipeline-sdk`) The SDK's HTTP client class is `MotionPipelineClient` (with `ConfigBuilder`, `JobBuilder`, and a `createClient()` convenience function). - **Full pipeline control**: Submit jobs, manage pipeline configurations (including YAML import/export), monitor progress, and retrieve results from Node.js applications. - **Job management**: List, filter, cancel, retry, and reprioritize jobs programmatically, plus batch operations. - **Progress streaming**: Subscribe to job progress over Server-Sent Events via `streamJobProgress()`, with per-event handler callbacks for job and stage events, heartbeats, and errors. - **Type-safe API**: Branded ID types and complete TypeScript definitions for all request parameters, configuration objects, and response shapes; every operation returns a discriminated `SDKResult`. ### Python SDK (`aja-motion-pipeline`) The Python distribution is `aja-motion-pipeline` (import package `aja_motion_pipeline`); the library directory is `libs/aja/motion-pipeline-sdk-python`. - **Sync and async clients**: Ships both a synchronous and an asynchronous client (`client.py`, `async_client.py`). - **Data science and Jupyter support**: Dedicated `data_science.py` and `jupyter.py` modules for notebook workflows. - **Batch and training helpers**: `batch.py` for batch submission and `training.py` for ML-training workflows. ### CLI Tools The CLI (`@aja/cli`) publishes the `aja` and `aja-motion` binaries. Each command targets a specific operator workflow: | Command | Description | | ------------- | ------------------------------------------------------------------------------------- | | `aja process` | Process a video or motion capture file through the pipeline with configurable options | | `aja convert` | Convert animation files between any supported formats | | `aja inspect` | Inspect a motion file and display metadata, quality metrics, and joint statistics | | `aja debug` | Run a single pipeline stage with verbose diagnostic output for debugging | | `aja config` | Create, list, and switch between named pipeline configuration profiles | | `aja jobs` | List, monitor, filter, and manage submitted processing jobs | | `aja health` | Check connectivity to all pipeline services and display their health status | --- ## 26. Cross-Domain Integration Integration adapters that deliver Aja's motion data into the rest of the Oshun ecosystem (`@aja/motion-integration`). Rather than allowing downstream domains to directly consume Aja's internal types, all cross-domain data exchange goes through these adapters — which translate Aja's data model into the format and protocol each consuming domain expects. This keeps Aja's pipeline decoupled from downstream concerns and makes the integration boundary explicit and testable. ### Shared Types and Adapter Interface - **Motion asset metadata**: Branded ID types (`MotionAssetId`, `MotionProjectId`, `MotionSessionId`, `MotionJobId`) plus structured metadata for motion assets — format, quality level, subject info, capture environment, and session quality metrics — ensuring consistent data contracts across all adapters. - **IntegrationAdapter interface**: A common interface that all domain adapters implement, covering connection lifecycle (`connect`, `disconnect`, `healthCheck`), asset publishing, event subscription, and pagination utilities — enabling callers to program against a single interface regardless of destination domain. - **Cross-domain references**: `CrossDomainReference` and `LinkedAsset` types record how a motion asset in Aja relates to counterpart assets in other domains, enabling bidirectional traceability. ### Yemaya Adapter (Film Production) Yemaya owns film production workflows. The integration boundary exists because Yemaya needs motion clips in DCC-compatible formats (Alembic, FBX) with production session context — metadata that belongs to Yemaya's production database, not to Aja's pipeline. - **Film asset delivery**: Publish finished motion clips in Alembic or FBX format directly into Yemaya's remote film production pipeline — automating the handoff from capture to compositing without file management. - **Production session tracking**: Associate motion capture sessions with specific Yemaya production projects, scenes, and shot identifiers for editorial traceability. - **Broadcast-grade quality gate**: Before delivery, the adapter checks that the motion meets Yemaya's minimum quality thresholds (jitter, foot slide, bone length variance) and rejects or flags substandard clips. ### Isis Adapter (3D Asset Creation) Isis owns 3D asset creation and management. The integration boundary keeps 3D asset assembly separate from motion processing: Aja delivers the animation and mesh together, and Isis integrates them into its content library. - **Rigged character export**: Deliver retargeted motion paired with the Aja avatar mesh to Isis for 3D asset assembly — Isis receives both the skeleton animation and the body mesh in a single handoff. - **USD and glTF delivery**: Export to Pixar USD (UsdSkel) and glTF/GLB formats for Isis's asset pipeline, preserving animation curves and skinning data. - **Motion asset registration**: Register exported motion assets in Isis's asset library with searchable metadata, making them discoverable inside the Isis content management system. ### Bellona Adapter (Build and Engine) Bellona owns build systems and artifact pipelines. The integration boundary enables deterministic, reproducible builds: motion artifacts are versioned and published to Bellona's registry, so downstream build jobs always consume a known-good version. - **Build artifact publishing**: After batch processing completes, publish motion output files to Bellona's artifact registry so that downstream build jobs can consume them deterministically. - **Job provenance tracking**: Record which Aja pipeline version, model configuration, and source video produced each artifact — enabling reproducible builds and auditable pipeline provenance. - **Quality report integration**: Attach Aja's per-clip quality reports (jitter score, MPJPE, plausibility) as build metadata, so build failures can be triggered when quality drops below configurable thresholds. ### Sophia Adapter (Research and Knowledge) Sophia owns research and knowledge management. The integration boundary lets motion-derived data enter Sophia's knowledge graph as citable, searchable research records rather than opaque files. - **Motion knowledge graph contributions**: Export motion clips, biomechanical measurements, and quality metrics into Sophia's knowledge graph, making motion data searchable and citable alongside other research assets. - **Research corpus management**: Register batches of processed motion as a research dataset with provenance, methodology, and source attribution — enabling academic and clinical research workflows built on Aja-processed data. - **Evidence linkage**: Link motion-derived biomechanical findings (e.g., range-of-motion measurements from the medical pipeline) to clinical evidence records in Sophia for integrated evidence review. --- ## 27. Planned Features Features planned in TODOS.md representing the next major expansion of Aja's capabilities. All items in this section are forward-looking — no corresponding code exists under `libs/aja/` or `apps/aja/`. ### CG Character Replacement from Video (`@aja/cg-replacement`) _(planned)_ This library enables replacing a live actor in a video with a CG character without a full VFX pipeline. - **Actor segmentation**: Isolate a person from video background using AI segmentation, enabling replacement of the live actor with a CG character in the original scene. - **Performance transfer to CG character rig**: Retarget captured pose directly to a CG character's custom rig, including facial performance and full-body motion. - **Automatic lighting estimation**: Estimate the lighting environment from the source video (direction, intensity, color) and apply it to the CG character renderer for photorealistic integration. - **CG-to-scene compositing**: Insert the rendered CG character back into the original scene with correct perspective, depth, and occlusion. - **Shadow and reflection synthesis**: Generate CG character shadows and reflections that match the real scene's lighting, completing the visual integration. ### AI Text-to-Animation (`@aja/generative-animation`) _(planned)_ Generate fully keyframed animation clips from natural language descriptions. - **Text-to-motion generation**: Describe a motion in natural language ("a person walks confidently, then stops and turns to look left") and receive a fully keyframed animation clip ready for export to BVH, FBX, or USD. - **Physics-informed generation**: Generated motion respects gravity, momentum, and contact — the character does not float or penetrate the floor. - **Multi-character interaction generation**: Generate two-person interactions (e.g., handshake, sparring) where both characters respond realistically to each other. - **Character-specific adaptation**: Motion is adapted to the target character's body proportions and physical capabilities. - **Natural language motion editing**: Edit existing animations with text instructions — "make the walk more tired," "slow down the arm swing" — without touching keyframes. ### AI Inbetweening and Motion Enhancement _(planned)_ - **Physics-aware inbetweening**: Given two keyframes, generate physically plausible interpolated motion that respects momentum and contact dynamics, rather than interpolating joint angles directly. - **Ballistic trajectory calculation**: For jumping and throwing motions, compute the correct parabolic arc from launch to landing. - **Contact-aware interpolation**: Foot contacts, hand grasps, and body-object interactions are preserved and enforced during inbetweening. - **Seamless loop generation**: Generate loopable animation clips with smooth transitions from the last frame back to the first. ### Oya Integration — Pose Estimation Streaming _(planned)_ - Real-time streaming of pose data from Aja to Oya (broadcast/streaming platform) for live motion-enhanced broadcasts, avatar-driven streaming, and synchronized recording with multi-camera systems. --- ## Neith Animation and Digital-Human Dependencies _(planned)_ This is a forward-looking cross-domain plan, not an implemented integration: no Aja library or service under `libs/aja/` or `apps/aja/` imports `@neith/*` or otherwise references the Neith domain. The intent is that Aja owns video-to-motion, pose estimation, retargeting, and animation transformation, while Neith owns workstation and engine primitives that Aja would use as capture/solve targets or downstream animation surfaces — across matchmove and tracking (Phase 153: Aja's 2D feature/pose tracking and ML-assisted tracking feed the `@neith/matchmove-*` camera/object solves), 2D-in-3D and digital-human surfaces (Phases 156, 162), and engine animation parity (Phases 171-174). The mechanism for any such integration is not yet present in code.