Date: 2026-04-27 Decision drivers:
aja-readiness-audit.md (V1-P2-0330),
aja-descope-decision.md (V1-P2-0331),
aja-runtime-strategy.md (V1-P2-0333). Status:
V1 posture locked (no GPU pipeline in V1). V2 candidate matrix recorded;
final selection deferred to a V2 ticket once the inputs identified below are
gathered.
Decision#
V1: no GPU pipeline. Aja ships only the CPU-only orchestration service in
V1, exactly as locked in V1-P2-0331 and V1-P2-0333. None of the GPU-related
libraries (batch-inference GPU memory paths, pipeline-parallelism,
model-optimization, neural-retargeting, human-mesh-recovery,
pose-lifting, depth-sensing, multi-view-reconstruction,
distributed-workers) are wired into a real runtime in V1.
V2: pick from the candidate matrix below once the four decision inputs are gathered. This document does not pick — picking now without volume, latency, and budget data would be premature.
Why no GPU pipeline in V1#
Recap from upstream tasks:
- The readiness audit found 17 explicit
"in production"markers and 124Math.random()calls in implementation source — load-bearing GPU paths are simulated. Lighting them up needs real models, drivers, and schedulers, none of which are present. - SMPL model weights, ONNX runtime selection, and a GPU operator are all out-of-monorepo decisions that the descope decision (V1-P2-0331) pulled out of V1's critical path.
- The K8s strategy (V1-P2-0333) ships zero GPU node pool. There is literally nowhere for GPU code to run in V1 even if it were finished.
Shipping V1 without a GPU pipeline is therefore consistent and honest: there is no half-built fallback the user could accidentally hit.
V2 candidate matrix#
Each candidate trades operational complexity against cost, latency, and lock-in. The aim of this section is to give the V2 owner a side-by-side comparison they can decide from once the input section below is filled in.
A — In-cluster CUDA on NVIDIA GPUs (via NVML / CUDA Runtime)#
- Stack: NVIDIA GPU Operator on the same K8s cluster as
svc-motion-pipeline. Inference workers run as Deployments / Jobs withnvidia.com/gpu: 1requests, scheduled to a GPU node pool. Health and utilization come from NVML exporters; the simulated paths inpipeline-parallelism/src/multi-gpu.ts(lines 89/219/313/413/442/498/689, see audit §2.2) get real implementations. - Inference runtime: ONNX Runtime with CUDA execution provider, or TensorRT for the model-optimization library. Either is broadly supported by the published model methods named in the Aja README (HMR 2.0, CLIFF, PyMAF-X, 4D-Humans, WHAM).
- Pros: unified observability and identity story (same cluster, same service
mesh). Already-typed
distributed-workersKubernetes client fits this directly. Lowest data-egress cost — video assets stay in region. - Cons: GPU nodes idle expensively. Cluster Autoscaler must support scale-to-zero for the GPU pool to be cost-effective. Driver upgrades and MIG / time-slicing decisions add ops surface area.
B — ROCm on AMD GPUs#
- Stack: Same shape as A but on AMD Instinct GPUs (MI200/MI300) via ROCm. Possible if the platform already operates AMD nodes for another domain.
- Pros: typically cheaper per-FLOP than NVIDIA in the same tier; diversification away from a single GPU vendor.
- Cons: smaller ecosystem of pretrained models that ship with ROCm builds. Several ML methods named in the Aja README do not have first-party ROCm support — bringing them up means recompiling kernels and validating numerics. Driver / runtime maturity (RCCL, MIOpen) lags CUDA. Most ONNX Runtime providers are CUDA-first.
- Verdict if no AMD GPUs already exist on platform: rule out. The porting cost is non-trivial and the V2 ML stack already targets CUDA.
C — Cloud-managed inference (SageMaker / Vertex AI / serverless GPU)#
- Stack: Inference moves out of the K8s cluster entirely.
svc-motion-pipelineposts inference requests over an SDK / REST boundary to the managed endpoint. Models are deployed via the cloud provider's model registry. The K8s footprint stays exactly the V1 CPU-only shape. - Pros: zero GPU ops on the platform team. Managed scale-to-zero. Per-request billing — predictable cost when load is bursty. Fastest way to get V2 inference live.
- Cons: cold-start latency on serverless tiers (10s–30s on first request) is incompatible with the coaching-overlay use case (interactive cues; latency budget likely <2 s). Vendor lock-in pulls model weights, monitoring, and identity into the chosen cloud. Data-egress costs scale with video volume. Cross-cloud federated identity needed if the primary K8s cluster lives elsewhere.
D — Hybrid (CPU control plane + sibling GPU plane)#
- Stack:
svc-motion-pipelinestays where V1 puts it (CPU K8s). GPU workers run as a separate VM pool / sibling cluster behind a private API. Thedistributed-workerslib's K8s client is rewritten as an HTTP / gRPC client. - Pros: decouples GPU-ops from the platform K8s. Lets the team iterate on the GPU plane without coordinating cluster releases. Easiest path to running on bare-metal GPU servers if cost demands it.
- Cons: extra network hop adds latency. Two control planes to observe. Health / timeout / retry semantics must be tuned across the boundary or transient blips become job failures.
Decision matrix#
| Criterion | A: in-cluster CUDA | B: ROCm | C: cloud-managed | D: hybrid |
|---|---|---|---|---|
Tracks the existing typed K8s client in distributed-workers |
Yes | Yes | No (HTTP) | No (HTTP) |
| Coaching-overlay latency budget achievable | Yes | Yes | Risky (cold start) | Yes if always-on |
| Idle-cost behavior at <10 % utilization | Bad | Bad | Best (scale-to-zero) | Mid |
| Vendor lock-in | Mid | Mid | High | Mid |
| Data-egress cost for video assets | Low | Low | High | Low–mid |
| Driver / runtime maturity for Aja-named methods (HMR/CLIFF/PyMAF-X) | Best | Mid | Best (managed) | Best |
| New ops surface for the platform team | Mid | Mid | Lowest | Highest |
(bold marks the candidate that wins on that row; ties allowed.)
No candidate dominates. The selection is sensitive to the four inputs below.
Decision inputs the V2 owner must collect#
These are the same inputs flagged in the runtime strategy doc, restated here as concrete asks:
- Sustained inference QPS estimate — what does the V2 product line actually need? Coaching-overlay traffic profile is the hardest case (high-frequency, low-latency). Demonstration plans are batchable.
- Per-endpoint latency budget — coaching-overlay needs sub-2 s P95; demonstration plans tolerate seconds. The latency target is the main driver against Candidate C's serverless tier.
- Existing GPU footprint on the platform — if Aja can ride an
already-funded GPU pool (e.g.
kalikarendering,freyavoice synthesis), Candidate A becomes nearly free incremental cost. If not, Candidates C and D become more attractive at low utilization. - Vendor / lock-in tolerance — Candidate C ties model weights and monitoring to a single cloud. If the broader org wants multi-cloud / on-prem optionality, Candidate C is excluded.
If all four inputs land "low volume, relaxed latency, no existing GPU pool, lock-in OK," the right call is C. If they land "high volume, tight latency, existing GPU pool, lock-in not OK," the right call is A. Anything in between favors D.
Cross-references for the V2 ticket#
libs/aja/distributed-workers/src/fault-tolerance.ts— simulated K8s health checks; replace with real client calls for Candidates A or D.libs/aja/pipeline-parallelism/src/multi-gpu.ts— 7 simulated CUDA / NVML paths; gate behind a feature flag and only enable for Candidate A.libs/aja/model-optimization/src/tensorrt-compiler.ts:267— simulated TensorRT engine builds; the real implementation is Candidate-A specific.libs/aja/human-mesh-recovery/src/smpl-recovery.ts:48— placeholder SMPL data; needs licensed SMPL/SMPL-X model files. This is an out-of-monorepo procurement step that V2 must complete regardless of the candidate chosen.libs/aja/neural-retargeting/src/inference.ts— typed ONNX interfaces but noonnxruntimedependency; V2 must add it (CUDA EP for Candidate A, CPU EP for fallback in D's control plane, or remove entirely if Candidate C is picked and the cloud provider's SDK is used directly).
Non-goals of this document#
- Picks no candidate. The V2 owner picks once decision inputs are in.
- Does not commit to specific instance families, GPU counts, or vendor contracts.
- Does not change V1 — V1's posture (no GPU pipeline) is already locked in V1-P2-0331 and V1-P2-0333.