# External Model Intelligence and Execution Providers This page covers the two halves of V1's generation supply chain: the **external model intelligence** layer (governed intake of Civitai and other external model sources into Isis-owned records) and the **execution provider** layer (the adapter library, the BFF executors that drive live providers, the realtime streaming path, and the render-farm scheduler). It is written for operators, platform engineers, and anyone deciding what credentials a deployment needs. It serves the audience-facing tiers described in [Generation Audience Tiers and Surface Boundaries](./generation-tiers-and-surfaces.md) and sits beneath the [Isis Generation Control](./isis-generation-control.md) substrate that makes "Isis is the only path" structurally true. The honest summary up front: the intake policy, the adapter library, the executors, the release-gate wiring, and the realtime seam are **real and unit-tested**. Every _live_ provider HTTP call is **env-gated and fail-closed by design**: with no key, the resolver returns `null` and the job fails closed with `provider_not_configured`. ## Civitai and external model intelligence External models never enter the runtime ungoverned. Every external source — Civitai first, others by the same policy — flows through an operator intake pipeline before any creator or contemplative surface can reference it. - **Intake policy** governs Civitai and other external model sources; unmanaged runtime use of an unreviewed external asset is blocked. - **Metadata ingestion** covers rights, license, creator attribution, content labels, preview assets, NSFW normalization, hashes, versions, lineage, recommended configuration, and compatibility notes. - **Review queues** cover intake, rights, license, compatibility, safety, preview approval, promotion, rejection, takedown, denylist, and staging recipes. - **Normalization** turns approved sources into Isis-owned model records held in Oshun-controlled storage — the raw external blob is never the runtime artifact. - **Operator review-queue UI** provides the queue list, item detail, license inspector, preview gallery, license-decision actions, staging-recipe binding, and audit trail. - **Operator search-and-import surface** wraps Civitai search _behind_ intake policy: search → preview → request import → review queue. There is no direct unmanaged download path. - **Managed-models browser** exposes only post-intake approved models to creator surfaces; raw Civitai access is operator-only (the `civitai-search` and `civitai-lora-hash-picker` surfaces, see [Generation Audience Tiers and Surface Boundaries](./generation-tiers-and-surfaces.md)). Tests cover metadata normalization, review-to-staging, license outcomes, compatibility notes, promotion decisions, bypass prevention, intake-bypass denial, license-decision audit, takedown cascade, and creator-view filtering. ## The provider adapter library The in-repo adapter catalog lives at `libs/isis/ai-providers/src/providers/`. It is real code with the following provider families (each a directory): | Directory | Purpose | | ------------------------------------------------ | ------------------------------------------------------------------- | | `image-generation` | Hosted text-to-image (`StabilityProvider`, SD3.5) | | `video-generation` | Hosted text/image-to-video (`LTXProvider`, fal.ai-hosted LTX-Video) | | `music-generation` | Music provider abstraction | | `tts` | Text-to-speech (ElevenLabs adapter) | | `three-d` | 3D generation | | `civitai` | Civitai model intelligence | | `comfy`, `comfy-cloud` | ComfyUI graph execution (self-hosted and cloud) | | `controlnet`, `instantid`, `ip-adapter` | Conditioning / identity / reference adapters | | `animation`, `video-processing` | Motion and post-processing | | `florence`, `llm`, `conversational-ai` | Vision-language, LLM, and conversational models | | `live-preview-streaming` | Streaming preview | | `model-registry`, `workflow-versioning`, `usage` | Registry, versioning, usage accounting | | `multi-gpu-orchestration` | Multi-GPU dispatch | A separate audio catalog lives at `libs/isis/audio-generation/src/generation/`, which contains `suno-provider.ts`, `udio-provider.ts`, `self-hosted.ts`, `sfx-provider.ts`, `music-generator.ts`, and `audio-analysis.ts`. > Accuracy correction: earlier doc text listed the music-generation provider set > as "MusicGen, Suno, Udio, Stable-Audio, custom-on-Comfy." The actual audio > generation directory has `suno-provider.ts`, `udio-provider.ts`, > `self-hosted.ts`, `sfx-provider.ts`, and `music-generator.ts` — there is > **no** `MusicGen` or `Stable-Audio` named provider file. The realtime music > path (below) wires Google **Magenta RT**, which the old list omitted entirely. ## The BFF generation executors The customer-facing execution path runs through the BFF executors at `apps/oshun/bff/src/generation/`. Each kind has an executor, a provider-env resolver, and tests: - `image-executor` / `image-provider-env` - `narration-executor` / `narration-provider-env` - `music-executor` / `music-provider-env` and `music-enqueue-executor` - `video-executor` / `video-provider-env` - `caption-dub-executor` / `caption-dub-provider-env` - `accessibility-pass-executor` - `explainer-executor` - `sky-briefing-executor` / `sky-briefing-provider-env` and `sky-briefing-enqueue-executor` - `curated-generation-executor` - `realtime-music-route` / `realtime-music-provider-env` Every executor routes its output through the **release-gate measurement** bridge (`provider-measurement.ts` → `buildReleaseMeasurement`), which builds a `CanonicalReleaseGateMeasurement` from the _real_ per-output governance signals. The two safety classifiers — `image-safety-classifier.ts` and `text-safety-classifier.ts` — supply the gate's `safetyScanScore`. The hard rule in `provider-measurement.ts` is **never default a signal to a passing constant**: an absent floor signal (`null`) maps to the most-blocking value (0), so an output with no real scan is faithfully _blocked_ by the release gate rather than released on fabricated evidence. When a classifier is absent or its scan fails, the score is `null` and the gate blocks — fail-closed, never an invented passing score. > Architecture-flow correction: the ARCHITECTURE request-flow mermaid labels the > execution provider as "ComfyUI · ElevenLabs · Suno." That is incomplete for > the live customer path. Image generation calls the **hosted Stability SD3.5** > endpoint (`StabilityProvider`, a text-to-image API with no ComfyUI graph > required), and video calls **fal.ai-hosted LTX-Video** (`LTXProvider`), not > ComfyUI directly. ComfyUI and RunPod are the _operator substrate_; the live > customer-facing executors call hosted HTTP providers (Stability / ElevenLabs / > Suno / fal.ai). The mermaid's provider label should be read as illustrative, > not exhaustive. ## Live providers and their env contracts Each live provider resolver is fail-closed: with no credentials it returns `null`, the executor injects `null`, and the job fails closed with `provider_not_configured`. The HTTP paths to Stability / ElevenLabs / Suno / LTX are, per their own source comments, "exercised at deploy time with a live key" — the env→provider wiring and request/result adaptation are unit-tested in CI, but the live HTTP call is not part of default CI. The deploy-time toggles that flip a surface from fail-closed to live: | Capability | Live provider | Required env | Optional env / defaults | | ----------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------- | | Image | Stability SD3.5 (`StabilityProvider`, `@isis/ai-providers/providers/image-generation`) | `OSHUN_STABILITY_*` API credentials | `OSHUN_STABILITY_MODEL` (default `sd3.5-large`) | | Image (alt) | OpenRouter unified Image API (`openrouter-image-provider.ts`; 30+ hosted models, FLUX.2 Pro default) — selected with `OSHUN_IMAGE_PROVIDER=openrouter`; no silent cross-vendor fallback | `OSHUN_OPENROUTER_API_KEY` (or `OPENROUTER_API_KEY`) | `OSHUN_OPENROUTER_IMAGE_MODEL` (default `black-forest-labs/flux-2-pro`) | | Voice / narration | ElevenLabs (resolved via `@psyche/voice-synthesis`, `createElevenLabsProvider`) | `OSHUN_ELEVENLABS_API_KEY` **and** `OSHUN_ELEVENLABS_VOICE_ID` | — | | Music (batch) | Suno (`SunoProvider`, `@isis/audio-generation/generation`) | `OSHUN_SUNO_API_KEY` | `OSHUN_SUNO_BASE_URL`; `OSHUN_SUNO_MODEL_VERSION` ∈ `v3` `v3.5` `v4` `v5` | | Video | fal.ai-hosted LTX-Video (`LTXProvider`, `@isis/ai-providers/providers/video-generation`) | fal.ai key | `OSHUN_VIDEO_MODEL` (default `ltx-video-2.0`) | | Music (realtime) | Magenta RT (`MagentaRtProvider`, `magentaRt` from `@euterpe/providers`) | `OSHUN_MRT2_ENABLED=true` **and** an on-device runtime that loads | — | Both narration credentials are required: the resolver in `narration-provider-env.ts` returns `null` unless _both_ `OSHUN_ELEVENLABS_API_KEY` and `OSHUN_ELEVENLABS_VOICE_ID` are present, so a deployment without a chosen voice never fabricates audio. The ElevenLabs adapter itself lives under `libs/isis/ai-providers/src/providers/tts/elevenlabs-provider.ts`, while the BFF voice path resolves the provider through `@psyche/voice-synthesis`'s `createElevenLabsProvider`. ### Image: aspect-ratio guardrails `image-provider-env.ts` validates the requested aspect ratio against the Stability SD3 preset set — `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `21:9`, `9:21`, `3:2`, `2:3` — and falls back to `1:1` for anything outside it. The adapter also emits the release-gate measurement from real governance signals and notes honestly that Stability does not sign C2PA, so the `c2pa.signed` flag rides through as whatever the provider returns (no fabricated provenance). ### Music: Suno wired, Udio present but not customer-path-wired The batch music resolver (`music-provider-env.ts`) wires **only Suno** today (`SunoProvider`, `OSHUN_SUNO_API_KEY`). The model version is validated against the set `{ v3, v3.5, v4, v5 }` via `resolveSunoModelVersion`, and `OSHUN_SUNO_BASE_URL` is an optional override. The Suno HTTP flow is submit → poll → download. > Contradiction reconciliation: deps§9's provider table lists both Suno **and** > Udio music adapters as V1-used. The `udio-provider.ts` adapter file exists in > the audio library, but the BFF music executor env resolver wires **only > Suno**. Udio is _library-present but not customer-path-wired_ — the docs imply > a parity the BFF wiring does not yet have. Treat Udio as an available adapter, > not a live customer path. Music outputs are also subject to per-workflow-class guardrails in `@isis/music-generation` (`guardrails.ts`), which enforce max duration, max file size, allowed MIME types (`audio/wav`, `audio/mpeg`, `audio/ogg`, and `audio/flac` for the `ritual` class), tempo BPM range, max intensity over `[0,1]`, and crescendo-rate limits — the contemplative-tone constraints that keep music safe for a customer in a crisis state. The library description is "provider abstraction, workflow classes, watermark + provenance (§24.8)." ### Realtime music is a distinct generation mode Unlike the batch pipeline (admit → enqueue → worker → `{ id, url? }`), the realtime music path is a long-lived, frame-level interactive session. Magenta RT ("MRT2") runs an **on-device runtime** (`@euterpe/realtime-gen`'s `NativeOnDeviceModelRuntime` / `SidecarOnDeviceRuntime`), and the `MagentaRtProvider` streams audio frame-by-frame. The route at `realtime-music-route.ts` is a Fastify WebSocket seam at `/v1/generation/realtime`: JSON `generation-control` envelopes flow _in_, binary PCM frames (little-endian f32, interleaved stereo, 48 kHz) flow _out_, and unified `stream.*` StreamEvents flow _out_ as JSON. It is fail-closed exactly like the Suno resolver: unless `OSHUN_MRT2_ENABLED=true` _and_ the native engine and weights load (the runtime reports `not_configured` otherwise), `realtime-music-provider-env.ts` returns `null` and the route serves a plain GET that returns **503 `mrt2_not_configured`** — it never opens a socket and never fakes a stream. This streaming mode is distinct from, and runs beside, the batch job queue. ## Execution substrate: ComfyUI, RunPod, and the render farm > 2026-07-03 substrate hardening: the 48-workflow canonical catalog is now > execution-verified by twin renderer contracts — the worker's Python renderer > (`docker/runpod/comfyui/catalog_renderer.py` + its unittest suite) and the > TypeScript twin (`@isis/workflows` `renderCatalogWorkflow` + spec) render > every workflow, every conditional toggle, with zero dangling links and valid > seeds. Worker images are flavor-parameterized (`MODEL_FLAVOR` sd15/sdxl/audio, > single `final` target so every image ships the handler), pin ComfyUI > `v0.27.0`, and install the exact node packs the catalog requires including > the in-repo `isis_comfyui_nodes` pack. The endpoint manifest generator > (`scripts/isis/generate_runpod_endpoint_image_manifests.mjs`) is a CI gate: > it validates every required node class and model against what images > actually provide and exits non-zero on drift. Full findings: > `V1_COMFYUI_SOTA_AUDIT_2026-07-03.md`. The operator execution substrate is governed and dashboarded: - **Workflow classes** with parameter guardrails, disallowed graph patterns, template versioning/diffing, portability checks, and prevention of ad hoc runtime dependencies. The operator graph editor binds a typed-node palette to the node registry, validates against the approved-class contract, supports save-as-class and semantic-version pinning, and ships a per-node visual diff / parameter-delta / portability / breaking-change report. - **Provider endpoints** support registration, health checks, region-aware routing, concurrency and cost controls, retry/backoff/dead-letter rules, secret rotation, failover, queue metrics, and release/outage/performance/audit tests. - **RunPod endpoint dashboard** surfaces per-endpoint queue depth, latency P50/P95/P99, error rate, cost/hour, region, GPU class, health badge, last-failover, and cold-start indicator, plus a cost & quota panel, a region-failover override UI, a secret-rotation surface, and a queue inspector with operator-only kill / requeue / reprioritize actions. The data model is `@isis/runpod-surface` — "endpoint registry, dashboard, cost/quota, queue inspector (§24.4)." ### The render-farm scheduler (`@oshun/render-farm`) The AAA-tier execution scheduler the surfaces gesture at is a real, separate library: `@oshun/render-farm` (`libs/oshun/render-farm`), imported by `apps/yemaya/studio-web`. It provides render-farm scheduling primitives — priority queues, worker-node capability matching, GPU-requirement matching, dependency execution, preemption, checkpoint/resume, cloud-burst, cost estimation, and dashboard snapshots. Its exported types include `RenderJob`, `RenderAssignment`, `RenderCheckpoint`, `RenderCloudBurstPlan` / `RenderCloudBurstDecision` / `RenderCloudBurstProvider`, `RenderCostEstimate`, `RenderGpuCapability` / `RenderGpuRequirement`, `PreemptionDecision`, `RenderDashboardSnapshot`, and `RenderQuotaBreach`, alongside the `RenderFarmScheduler` (`createRenderFarmScheduler`). This is the named execution scheduler that backs the AAA `gpu-worker`, `multi-gpu-orchestration`, and `runpod-region-selector` surfaces. ### The autonomous creative orchestrator (`@oshun/creative-orchestrator`) For curated and contemplative generation that begins from a brief rather than a single prompt, `@oshun/creative-orchestrator` (`libs/oshun/creative-orchestrator`) is the real, BFF-wired connective tissue (used by `apps/oshun/bff/src/agentic/creative-generator-tools.ts` and `agentic-governance-gate.ts`, plus `libs/yemaya` orchestration and agents). It implements: - `decomposeBrief` — turns a brief into a schema-validated, acyclic `CreativePlan` DAG (validated against `CREATIVE_PLAN_SCHEMA`, with `validateDagStructure` / `detectCycle` / `topologicalOrder`), built on `@oshun/ai/agent-loop`'s `runStructuredOutput`. - `routePlan` / `CreativeOrchestrator` — governed dispatch of plan nodes to domain generators (`GeneratorRegistry`), gated by `BudgetGovernanceGate`. - `reviseArtifact` — a bounded generate → critique → revise (Reflexion) loop with pluggable critics (`createMetricCritic`, `createLlmJudgeCritic`, `createContentEvalCritic`). - `createYemayaAgentGenerator` / `createMetisNarrator` — adapters that wire Yemaya specialized agents and Metis narration into the plan. It is **fail-loud throughout** — it never fabricates a plan or an artifact, and the LLM-dependent steps fail when no provider is configured rather than inventing output. ## LoRA training, model merging, and tuning LoRA training and model merging exist as data models and surfaces (never on the contemplative product). The library `@isis/lora-training-surface` is described as "LoRA training queue + model merging + tuning rehearsal + quality scoring + lineage (§24.5)" and covers five surfaces: the training-queue UI (dataset upload with rights attestation, run config, progress timeline, per-checkpoint sample previews, cost estimate, abort, promotion-to-managed-models gate); the model-merging surface (component selection, merge ratio, A/B preview against a fixture set, save-as-managed-model with a provenance bundle); mandatory tuning rehearsal (fixture-set evaluation whose eval-score delta blocks promotion below baseline); model-quality scorecards with drift/deprecation indicators; and the training-run lineage tree (dataset → run → checkpoint → merge → managed-model edges carrying provenance, rights, and consent IDs). The data models and surfaces are real; live training _execution_ (and Gaussian splatting and the full 3D pipeline) is gated on deploy-time GPU substrate and not verified end-to-end in CI. Backlog references: RunPod operator surface §24.4; LoRA training §24.5; music generation §24.8; output gallery §24.10. The external-provider table is deps§9, which lists RunPod, ComfyUI, Civitai, SDXL, Flux, ElevenLabs, Suno, Udio, self-hosted audio, and 3D generation, and notes that no `elevenlabs` SDK is pinned (the adapter is a direct HTTP client). ## Related - [Isis Generation Control](./isis-generation-control.md) - [Generation Audience Tiers and Surface Boundaries](./generation-tiers-and-surfaces.md) - [Creator Surfaces, Voice, Music, and 3D Generation](./creator-voice-music-3d.md) - [Output Gallery, Lineage, Branch, and Replay](./output-gallery-lineage.md) - [Sophia Grounding](./sophia-grounding.md) - [Review, Compliance, and Trust & Safety](./review-trust-safety.md) - [Agent Registry, Job Orchestration, and Multi-Agent Plans](./agentic-registry-jobs-plans.md) - [Subsystem Glossary](./glossary.md) - [Architecture, Platform Foundations, and Security](./platform-foundations-and-security.md) - Hub: [../features.md](../features.md)