Eve library

Aja GPU Pipeline Strategy (V1-P2-0334)

Guide

Revision 1

On this page

Date: 2026-04-27 Decision drivers: aja-readiness-audit.md (V1-P2-0330), aja-descope-decision.md (V1-P2-0331), aja-runtime-strategy.md (V1-P2-0333). Status: V1 posture locked (no GPU pipeline in V1). V2 candidate matrix recorded; final selection deferred to a V2 ticket once the inputs identified below are gathered.

Decision#

V1: no GPU pipeline. Aja ships only the CPU-only orchestration service in V1, exactly as locked in V1-P2-0331 and V1-P2-0333. None of the GPU-related libraries (batch-inference GPU memory paths, pipeline-parallelism, model-optimization, neural-retargeting, human-mesh-recovery, pose-lifting, depth-sensing, multi-view-reconstruction, distributed-workers) are wired into a real runtime in V1.

V2: pick from the candidate matrix below once the four decision inputs are gathered. This document does not pick — picking now without volume, latency, and budget data would be premature.

Why no GPU pipeline in V1#

Recap from upstream tasks:

  • The readiness audit found 17 explicit "in production" markers and 124 Math.random() calls in implementation source — load-bearing GPU paths are simulated. Lighting them up needs real models, drivers, and schedulers, none of which are present.
  • SMPL model weights, ONNX runtime selection, and a GPU operator are all out-of-monorepo decisions that the descope decision (V1-P2-0331) pulled out of V1's critical path.
  • The K8s strategy (V1-P2-0333) ships zero GPU node pool. There is literally nowhere for GPU code to run in V1 even if it were finished.

Shipping V1 without a GPU pipeline is therefore consistent and honest: there is no half-built fallback the user could accidentally hit.

V2 candidate matrix#

Each candidate trades operational complexity against cost, latency, and lock-in. The aim of this section is to give the V2 owner a side-by-side comparison they can decide from once the input section below is filled in.

A — In-cluster CUDA on NVIDIA GPUs (via NVML / CUDA Runtime)#

  • Stack: NVIDIA GPU Operator on the same K8s cluster as svc-motion-pipeline. Inference workers run as Deployments / Jobs with nvidia.com/gpu: 1 requests, scheduled to a GPU node pool. Health and utilization come from NVML exporters; the simulated paths in pipeline-parallelism/src/multi-gpu.ts (lines 89/219/313/413/442/498/689, see audit §2.2) get real implementations.
  • Inference runtime: ONNX Runtime with CUDA execution provider, or TensorRT for the model-optimization library. Either is broadly supported by the published model methods named in the Aja README (HMR 2.0, CLIFF, PyMAF-X, 4D-Humans, WHAM).
  • Pros: unified observability and identity story (same cluster, same service mesh). Already-typed distributed-workers Kubernetes client fits this directly. Lowest data-egress cost — video assets stay in region.
  • Cons: GPU nodes idle expensively. Cluster Autoscaler must support scale-to-zero for the GPU pool to be cost-effective. Driver upgrades and MIG / time-slicing decisions add ops surface area.

B — ROCm on AMD GPUs#

  • Stack: Same shape as A but on AMD Instinct GPUs (MI200/MI300) via ROCm. Possible if the platform already operates AMD nodes for another domain.
  • Pros: typically cheaper per-FLOP than NVIDIA in the same tier; diversification away from a single GPU vendor.
  • Cons: smaller ecosystem of pretrained models that ship with ROCm builds. Several ML methods named in the Aja README do not have first-party ROCm support — bringing them up means recompiling kernels and validating numerics. Driver / runtime maturity (RCCL, MIOpen) lags CUDA. Most ONNX Runtime providers are CUDA-first.
  • Verdict if no AMD GPUs already exist on platform: rule out. The porting cost is non-trivial and the V2 ML stack already targets CUDA.

C — Cloud-managed inference (SageMaker / Vertex AI / serverless GPU)#

  • Stack: Inference moves out of the K8s cluster entirely. svc-motion-pipeline posts inference requests over an SDK / REST boundary to the managed endpoint. Models are deployed via the cloud provider's model registry. The K8s footprint stays exactly the V1 CPU-only shape.
  • Pros: zero GPU ops on the platform team. Managed scale-to-zero. Per-request billing — predictable cost when load is bursty. Fastest way to get V2 inference live.
  • Cons: cold-start latency on serverless tiers (10s–30s on first request) is incompatible with the coaching-overlay use case (interactive cues; latency budget likely <2 s). Vendor lock-in pulls model weights, monitoring, and identity into the chosen cloud. Data-egress costs scale with video volume. Cross-cloud federated identity needed if the primary K8s cluster lives elsewhere.

D — Hybrid (CPU control plane + sibling GPU plane)#

  • Stack: svc-motion-pipeline stays where V1 puts it (CPU K8s). GPU workers run as a separate VM pool / sibling cluster behind a private API. The distributed-workers lib's K8s client is rewritten as an HTTP / gRPC client.
  • Pros: decouples GPU-ops from the platform K8s. Lets the team iterate on the GPU plane without coordinating cluster releases. Easiest path to running on bare-metal GPU servers if cost demands it.
  • Cons: extra network hop adds latency. Two control planes to observe. Health / timeout / retry semantics must be tuned across the boundary or transient blips become job failures.

Decision matrix#

Criterion A: in-cluster CUDA B: ROCm C: cloud-managed D: hybrid
Tracks the existing typed K8s client in distributed-workers Yes Yes No (HTTP) No (HTTP)
Coaching-overlay latency budget achievable Yes Yes Risky (cold start) Yes if always-on
Idle-cost behavior at <10 % utilization Bad Bad Best (scale-to-zero) Mid
Vendor lock-in Mid Mid High Mid
Data-egress cost for video assets Low Low High Low–mid
Driver / runtime maturity for Aja-named methods (HMR/CLIFF/PyMAF-X) Best Mid Best (managed) Best
New ops surface for the platform team Mid Mid Lowest Highest

(bold marks the candidate that wins on that row; ties allowed.)

No candidate dominates. The selection is sensitive to the four inputs below.

Decision inputs the V2 owner must collect#

These are the same inputs flagged in the runtime strategy doc, restated here as concrete asks:

  1. Sustained inference QPS estimate — what does the V2 product line actually need? Coaching-overlay traffic profile is the hardest case (high-frequency, low-latency). Demonstration plans are batchable.
  2. Per-endpoint latency budget — coaching-overlay needs sub-2 s P95; demonstration plans tolerate seconds. The latency target is the main driver against Candidate C's serverless tier.
  3. Existing GPU footprint on the platform — if Aja can ride an already-funded GPU pool (e.g. kalika rendering, freya voice synthesis), Candidate A becomes nearly free incremental cost. If not, Candidates C and D become more attractive at low utilization.
  4. Vendor / lock-in tolerance — Candidate C ties model weights and monitoring to a single cloud. If the broader org wants multi-cloud / on-prem optionality, Candidate C is excluded.

If all four inputs land "low volume, relaxed latency, no existing GPU pool, lock-in OK," the right call is C. If they land "high volume, tight latency, existing GPU pool, lock-in not OK," the right call is A. Anything in between favors D.

Cross-references for the V2 ticket#

  • libs/aja/distributed-workers/src/fault-tolerance.ts — simulated K8s health checks; replace with real client calls for Candidates A or D.
  • libs/aja/pipeline-parallelism/src/multi-gpu.ts — 7 simulated CUDA / NVML paths; gate behind a feature flag and only enable for Candidate A.
  • libs/aja/model-optimization/src/tensorrt-compiler.ts:267 — simulated TensorRT engine builds; the real implementation is Candidate-A specific.
  • libs/aja/human-mesh-recovery/src/smpl-recovery.ts:48 — placeholder SMPL data; needs licensed SMPL/SMPL-X model files. This is an out-of-monorepo procurement step that V2 must complete regardless of the candidate chosen.
  • libs/aja/neural-retargeting/src/inference.ts — typed ONNX interfaces but no onnxruntime dependency; V2 must add it (CUDA EP for Candidate A, CPU EP for fallback in D's control plane, or remove entirely if Candidate C is picked and the cloud provider's SDK is used directly).

Non-goals of this document#

  • Picks no candidate. The V2 owner picks once decision inputs are in.
  • Does not commit to specific instance families, GPU counts, or vendor contracts.
  • Does not change V1 — V1's posture (no GPU pipeline) is already locked in V1-P2-0331 and V1-P2-0333.