# Infra — Systems Deep Dive

> The `apps/infra/` area: two deployable Nx applications that operate Oshun's
> RunPod Serverless GPU plane — one that dispatches GPU jobs and one that
> monitors the endpoints those jobs run on.

## What this area is

`apps/infra/` is the operational glue around Oshun's off-box GPU compute. Heavy
generative work (image, video, audio, motion-capture, model training/inference)
is run on **RunPod Serverless** endpoints rather than on the application
servers, and this directory holds the two long-running processes that stand
between the platform and that external GPU fleet. Both are Nx
`projectType: "application"` nodes tagged `scope:shared` / `type:app`, built
with `tsup`, and packaged as standalone services (the dispatcher even ships a
`Dockerfile` and an ECS/Fargate `deploy/` bundle).

The two apps split cleanly along the **control-plane vs. observability-plane**
line. `gpu-dispatcher-worker` (`apps/infra/gpu-dispatcher`) is the control path:
it pulls GPU job requests off a Redis-backed queue, submits them to RunPod,
waits for completion, normalises the results, and fires result callbacks.
`runpod-metrics-collector` (`apps/infra/runpod-metrics`) is the observability
path: it polls the same RunPod endpoints for health and worker state and
publishes derived metrics (utilisation, queue saturation, failure rate,
estimated cost) to AWS CloudWatch.

They are siblings, not a dependency pair — neither app imports the other.
Instead they share a common substrate of `libs/shared` packages: both build on
`@oshun/runpod-client` (the typed RunPod SDK), `@oshun/logging`, and
`@oshun/config`. The dispatcher additionally composes `@oshun/queue` (the
BullMQ-style worker + dead-letter-queue), `@oshun/gpu-dispatcher` (endpoint
selection / job lifecycle), `@oshun/storage`, and `@oshun/health`. Those shared
dependencies are declared both as `implicitDependencies` in each `project.json`
and as `workspace:*` deps in each `package.json`.

## How it fits the wider system

These are leaf deployables at the bottom of the runtime topology: domain
services and the BFF enqueue GPU work (onto the `gpu-jobs` queue the dispatcher
drains) and consume the results via the callback URLs the dispatcher posts back.
Nothing in the monorepo imports `apps/infra/*` as a library — they are
processes, not packages (the dispatcher's `package.json` is `private` with no
`exports`; the metrics collector does export its `RunPodMetricsCollector` class,
but its real role is the `main()` entry point run as an ECS task or Lambda). The
boundary they own is the seam between Oshun's TypeScript runtime and RunPod's
GPU workers: the dispatcher translates internal job envelopes into RunPod calls
and back, and the collector translates RunPod health into CloudWatch telemetry.
Walk the "used by" edges below to see exactly which shared libraries each one
composes.

## Entity reference

### gpu-dispatcher-worker

A queue-based GPU job worker (`apps/infra/gpu-dispatcher`, package
`gpu-dispatcher-worker`, description "GPU job dispatcher worker for RunPod
Serverless"). Its entry point `src/index.ts` loads a Zod-validated environment
config (`src/config.ts` — Redis, queue, RunPod endpoints, storage, worker, and
metrics sections), constructs the `GpuWorker`, wires its lifecycle events to
structured logging, and installs SIGTERM/SIGINT graceful-shutdown handlers. The
core is `src/worker.ts`: `GpuWorker` (an `EventEmitter`) drains jobs from a
`@oshun/queue` `createWorker`, dispatches each to RunPod via
`createGpuDispatcher`, polls for completion with `dispatcher.waitForJob`,
converts the dispatcher result into the worker's `GpuJobResult` shape (output
type/MIME inference, execution metrics, RunPod metadata in `src/types.ts`),
emits progress at 10/30/90/100%, and posts an HMAC-able result callback via
`fetch`. Failures flow through a dead-letter queue (`createDeadLetterQueue`,
30-day retention) with a `getFailureReason` classifier (`timeout` / `cancelled`
/ `retry-exhausted` / `processing-error`). This is real, domain-specific
infrastructure code, not a scaffold, and it ships a multi-stage `Dockerfile`
plus an ECS/Fargate `deploy/` bundle (`task-definition.json`,
`service-definition.json`, Terraform).

One honest caveat worth recording: `src/callbacks.ts` defines a complete,
unit-tested `CallbackHandler` — an HTTP server for inbound RunPod webhooks
(`/webhook/runpod` + `/health`) with timing-safe HMAC signature validation and a
genuinely careful SSRF guard on tenant-supplied callback URLs
(`validateCallbackUrl` blocks `localhost`, RFC1918, link-local, cloud-metadata,
and IPv4-mapped-IPv6 addresses against a host-suffix allowlist). That handler is
the surface the `deploy/task-definition.json` health check
(`curl http://localhost:8080/health`) and `WEBHOOK_SECRET` expect — but the
current `src/index.ts` entry point starts only the queue worker and does **not**
instantiate `CallbackHandler`, so the webhook/HTTP path exists and is tested but
is not yet wired into the running process. Similarly, the `WorkerHealthStatus` /
`EndpointHealthStatus` types in `src/types.ts` are declared but not yet produced
by the worker.

### runpod-metrics-collector

A RunPod-to-CloudWatch metrics collector (`apps/infra/runpod-metrics`, package
`runpod-metrics-collector`, description "RunPod metrics collector for CloudWatch
integration"). The whole implementation lives in `src/index.ts`:
`RunPodMetricsCollector` polls each configured endpoint's health through
`@oshun/runpod-client`'s `health()`, computes derived metrics — worker
utilisation, queue saturation (queued jobs vs. available capacity), failure
rate, and an estimated hourly cost from running-worker count × a per-second GPU
price (`DEFAULT_GPU_COST_PER_SECOND = 0.00044`, ≈ $1.58/hr) — and pushes both
per-endpoint and cross-endpoint aggregate `MetricDatum` to AWS CloudWatch via
`@aws-sdk/client-cloudwatch`'s `PutMetricDataCommand`, batching at 500 metrics
per request. It runs in two modes selected by `RUN_MODE`: `continuous`
(`setInterval` on a configurable collection interval, default 60s, for an ECS
task) or `oneshot` (collect once and exit, for a scheduled Lambda). `loadConfig`
assembles the endpoint list from `RUNPOD_ENDPOINT_IDS` plus named feature-flag
endpoints (`RUNPOD_COMFYUI_ENDPOINT_ID`, `_SD_`, `_FLUX_`) and throws if none
are configured. This is real, working code with domain-specific metric formulas;
cost/detailed metrics are gated behind `enableCostTracking` / `detailedMetrics`
flags. It is an ESM package (`"type": "module"`) and ships no Dockerfile or
`deploy/` of its own — only `src/` plus tests (`src/index.spec.ts`).