External Model Intelligence and Execution Providers

This page covers the two halves of V1's generation supply chain: the external model intelligence layer (governed intake of Civitai and other external model sources into Isis-owned records) and the execution provider layer (the adapter library, the BFF executors that drive live providers, the realtime streaming path, and the render-farm scheduler). It is written for operators, platform engineers, and anyone deciding what credentials a deployment needs. It serves the audience-facing tiers described in Generation Audience Tiers and Surface Boundaries and sits beneath the Isis Generation Control substrate that makes "Isis is the only path" structurally true. The honest summary up front: the intake policy, the adapter library, the executors, the release-gate wiring, and the realtime seam are real and unit-tested. Every live provider HTTP call is env-gated and fail-closed by design: with no key, the resolver returns null and the job fails closed with provider_not_configured.

Civitai and external model intelligence#

External models never enter the runtime ungoverned. Every external source — Civitai first, others by the same policy — flows through an operator intake pipeline before any creator or contemplative surface can reference it.

• Intake policy governs Civitai and other external model sources; unmanaged runtime use of an unreviewed external asset is blocked.
• Metadata ingestion covers rights, license, creator attribution, content labels, preview assets, NSFW normalization, hashes, versions, lineage, recommended configuration, and compatibility notes.
• Review queues cover intake, rights, license, compatibility, safety, preview approval, promotion, rejection, takedown, denylist, and staging recipes.
• Normalization turns approved sources into Isis-owned model records held in Oshun-controlled storage — the raw external blob is never the runtime artifact.
• Operator review-queue UI provides the queue list, item detail, license inspector, preview gallery, license-decision actions, staging-recipe binding, and audit trail.
• Operator search-and-import surface wraps Civitai search behind intake policy: search → preview → request import → review queue. There is no direct unmanaged download path.
• Managed-models browser exposes only post-intake approved models to creator surfaces; raw Civitai access is operator-only (the civitai-search and civitai-lora-hash-picker surfaces, see Generation Audience Tiers and Surface Boundaries).

Tests cover metadata normalization, review-to-staging, license outcomes, compatibility notes, promotion decisions, bypass prevention, intake-bypass denial, license-decision audit, takedown cascade, and creator-view filtering.

The provider adapter library#

The in-repo adapter catalog lives at libs/isis/ai-providers/src/providers/. It is real code with the following provider families (each a directory):

A separate audio catalog lives at libs/isis/audio-generation/src/generation/, which contains suno-provider.ts, udio-provider.ts, self-hosted.ts, sfx-provider.ts, music-generator.ts, and audio-analysis.ts.

Accuracy correction: earlier doc text listed the music-generation provider set as "MusicGen, Suno, Udio, Stable-Audio, custom-on-Comfy." The actual audio generation directory has suno-provider.ts, udio-provider.ts, self-hosted.ts, sfx-provider.ts, and music-generator.ts — there is no MusicGen or Stable-Audio named provider file. The realtime music path (below) wires Google Magenta RT, which the old list omitted entirely.

The BFF generation executors#

The customer-facing execution path runs through the BFF executors at apps/oshun/bff/src/generation/. Each kind has an executor, a provider-env resolver, and tests:

• image-executor / image-provider-env
• narration-executor / narration-provider-env
• music-executor / music-provider-env and music-enqueue-executor
• video-executor / video-provider-env
• caption-dub-executor / caption-dub-provider-env
• accessibility-pass-executor
• explainer-executor
• sky-briefing-executor / sky-briefing-provider-env and sky-briefing-enqueue-executor
• curated-generation-executor
• realtime-music-route / realtime-music-provider-env

Every executor routes its output through the release-gate measurement bridge (provider-measurement.ts → buildReleaseMeasurement), which builds a CanonicalReleaseGateMeasurement from the real per-output governance signals. The two safety classifiers — image-safety-classifier.ts and text-safety-classifier.ts — supply the gate's safetyScanScore. The hard rule in provider-measurement.ts is never default a signal to a passing constant: an absent floor signal (null) maps to the most-blocking value (0), so an output with no real scan is faithfully blocked by the release gate rather than released on fabricated evidence. When a classifier is absent or its scan fails, the score is null and the gate blocks — fail-closed, never an invented passing score.

Architecture-flow correction: the ARCHITECTURE request-flow mermaid labels the execution provider as "ComfyUI · ElevenLabs · Suno." That is incomplete for the live customer path. Image generation calls the hosted Stability SD3.5 endpoint (StabilityProvider, a text-to-image API with no ComfyUI graph required), and video calls fal.ai-hosted LTX-Video (LTXProvider), not ComfyUI directly. ComfyUI and RunPod are the operator substrate; the live customer-facing executors call hosted HTTP providers (Stability / ElevenLabs / Suno / fal.ai). The mermaid's provider label should be read as illustrative, not exhaustive.

Live providers and their env contracts#

Each live provider resolver is fail-closed: with no credentials it returns null, the executor injects null, and the job fails closed with provider_not_configured. The HTTP paths to Stability / ElevenLabs / Suno / LTX are, per their own source comments, "exercised at deploy time with a live key" — the env→provider wiring and request/result adaptation are unit-tested in CI, but the live HTTP call is not part of default CI. The deploy-time toggles that flip a surface from fail-closed to live:

Both narration credentials are required: the resolver in narration-provider-env.ts returns null unless both OSHUN_ELEVENLABS_API_KEY and OSHUN_ELEVENLABS_VOICE_ID are present, so a deployment without a chosen voice never fabricates audio. The ElevenLabs adapter itself lives under libs/isis/ai-providers/src/providers/tts/elevenlabs-provider.ts, while the BFF voice path resolves the provider through @psyche/voice-synthesis's createElevenLabsProvider.

Image: aspect-ratio guardrails#

image-provider-env.ts validates the requested aspect ratio against the Stability SD3 preset set — 1:1, 16:9, 9:16, 4:3, 3:4, 21:9, 9:21, 3:2, 2:3 — and falls back to 1:1 for anything outside it. The adapter also emits the release-gate measurement from real governance signals and notes honestly that Stability does not sign C2PA, so the c2pa.signed flag rides through as whatever the provider returns (no fabricated provenance).

Music: Suno wired, Udio present but not customer-path-wired#

The batch music resolver (music-provider-env.ts) wires only Suno today (SunoProvider, OSHUN_SUNO_API_KEY). The model version is validated against the set { v3, v3.5, v4, v5 } via resolveSunoModelVersion, and OSHUN_SUNO_BASE_URL is an optional override. The Suno HTTP flow is submit → poll → download.

Contradiction reconciliation: deps§9's provider table lists both Suno and Udio music adapters as V1-used. The udio-provider.ts adapter file exists in the audio library, but the BFF music executor env resolver wires only Suno. Udio is library-present but not customer-path-wired — the docs imply a parity the BFF wiring does not yet have. Treat Udio as an available adapter, not a live customer path.

Music outputs are also subject to per-workflow-class guardrails in @isis/music-generation (guardrails.ts), which enforce max duration, max file size, allowed MIME types (audio/wav, audio/mpeg, audio/ogg, and audio/flac for the ritual class), tempo BPM range, max intensity over [0,1], and crescendo-rate limits — the contemplative-tone constraints that keep music safe for a customer in a crisis state. The library description is "provider abstraction, workflow classes, watermark + provenance (§24.8)."

Realtime music is a distinct generation mode#

Unlike the batch pipeline (admit → enqueue → worker → { id, url? }), the realtime music path is a long-lived, frame-level interactive session. Magenta RT ("MRT2") runs an on-device runtime (@euterpe/realtime-gen's NativeOnDeviceModelRuntime / SidecarOnDeviceRuntime), and the MagentaRtProvider streams audio frame-by-frame. The route at realtime-music-route.ts is a Fastify WebSocket seam at /v1/generation/realtime: JSON generation-control envelopes flow in, binary PCM frames (little-endian f32, interleaved stereo, 48 kHz) flow out, and unified stream.* StreamEvents flow out as JSON. It is fail-closed exactly like the Suno resolver: unless OSHUN_MRT2_ENABLED=true and the native engine and weights load (the runtime reports not_configured otherwise), realtime-music-provider-env.ts returns null and the route serves a plain GET that returns 503 mrt2_not_configured — it never opens a socket and never fakes a stream. This streaming mode is distinct from, and runs beside, the batch job queue.

Execution substrate: ComfyUI, RunPod, and the render farm#

2026-07-03 substrate hardening: the 48-workflow canonical catalog is now execution-verified by twin renderer contracts — the worker's Python renderer (docker/runpod/comfyui/catalog_renderer.py + its unittest suite) and the TypeScript twin (@isis/workflows renderCatalogWorkflow + spec) render every workflow, every conditional toggle, with zero dangling links and valid seeds. Worker images are flavor-parameterized (MODEL_FLAVOR sd15/sdxl/audio, single final target so every image ships the handler), pin ComfyUI v0.27.0, and install the exact node packs the catalog requires including the in-repo isis_comfyui_nodes pack. The endpoint manifest generator (scripts/isis/generate_runpod_endpoint_image_manifests.mjs) is a CI gate: it validates every required node class and model against what images actually provide and exits non-zero on drift. Full findings: V1_COMFYUI_SOTA_AUDIT_2026-07-03.md.

The operator execution substrate is governed and dashboarded:

• Workflow classes with parameter guardrails, disallowed graph patterns, template versioning/diffing, portability checks, and prevention of ad hoc runtime dependencies. The operator graph editor binds a typed-node palette to the node registry, validates against the approved-class contract, supports save-as-class and semantic-version pinning, and ships a per-node visual diff / parameter-delta / portability / breaking-change report.
• Provider endpoints support registration, health checks, region-aware routing, concurrency and cost controls, retry/backoff/dead-letter rules, secret rotation, failover, queue metrics, and release/outage/performance/audit tests.
• RunPod endpoint dashboard surfaces per-endpoint queue depth, latency P50/P95/P99, error rate, cost/hour, region, GPU class, health badge, last-failover, and cold-start indicator, plus a cost & quota panel, a region-failover override UI, a secret-rotation surface, and a queue inspector with operator-only kill / requeue / reprioritize actions. The data model is @isis/runpod-surface — "endpoint registry, dashboard, cost/quota, queue inspector (§24.4)."

The render-farm scheduler (@oshun/render-farm)#

The AAA-tier execution scheduler the surfaces gesture at is a real, separate library: @oshun/render-farm (libs/oshun/render-farm), imported by apps/yemaya/studio-web. It provides render-farm scheduling primitives — priority queues, worker-node capability matching, GPU-requirement matching, dependency execution, preemption, checkpoint/resume, cloud-burst, cost estimation, and dashboard snapshots. Its exported types include RenderJob, RenderAssignment, RenderCheckpoint, RenderCloudBurstPlan / RenderCloudBurstDecision / RenderCloudBurstProvider, RenderCostEstimate, RenderGpuCapability / RenderGpuRequirement, PreemptionDecision, RenderDashboardSnapshot, and RenderQuotaBreach, alongside the RenderFarmScheduler (createRenderFarmScheduler). This is the named execution scheduler that backs the AAA gpu-worker, multi-gpu-orchestration, and runpod-region-selector surfaces.

The autonomous creative orchestrator (@oshun/creative-orchestrator)#

For curated and contemplative generation that begins from a brief rather than a single prompt, @oshun/creative-orchestrator (libs/oshun/creative-orchestrator) is the real, BFF-wired connective tissue (used by apps/oshun/bff/src/agentic/creative-generator-tools.ts and agentic-governance-gate.ts, plus libs/yemaya orchestration and agents). It implements:

• decomposeBrief — turns a brief into a schema-validated, acyclic CreativePlan DAG (validated against CREATIVE_PLAN_SCHEMA, with validateDagStructure / detectCycle / topologicalOrder), built on @oshun/ai/agent-loop's runStructuredOutput.
• routePlan / CreativeOrchestrator — governed dispatch of plan nodes to domain generators (GeneratorRegistry), gated by BudgetGovernanceGate.
• reviseArtifact — a bounded generate → critique → revise (Reflexion) loop with pluggable critics (createMetricCritic, createLlmJudgeCritic, createContentEvalCritic).
• createYemayaAgentGenerator / createMetisNarrator — adapters that wire Yemaya specialized agents and Metis narration into the plan.

It is fail-loud throughout — it never fabricates a plan or an artifact, and the LLM-dependent steps fail when no provider is configured rather than inventing output.

LoRA training, model merging, and tuning#

LoRA training and model merging exist as data models and surfaces (never on the contemplative product). The library @isis/lora-training-surface is described as "LoRA training queue + model merging + tuning rehearsal + quality scoring + lineage (§24.5)" and covers five surfaces: the training-queue UI (dataset upload with rights attestation, run config, progress timeline, per-checkpoint sample previews, cost estimate, abort, promotion-to-managed-models gate); the model-merging surface (component selection, merge ratio, A/B preview against a fixture set, save-as-managed-model with a provenance bundle); mandatory tuning rehearsal (fixture-set evaluation whose eval-score delta blocks promotion below baseline); model-quality scorecards with drift/deprecation indicators; and the training-run lineage tree (dataset → run → checkpoint → merge → managed-model edges carrying provenance, rights, and consent IDs). The data models and surfaces are real; live training execution (and Gaussian splatting and the full 3D pipeline) is gated on deploy-time GPU substrate and not verified end-to-end in CI.

Backlog references: RunPod operator surface §24.4; LoRA training §24.5; music generation §24.8; output gallery §24.10. The external-provider table is deps§9, which lists RunPod, ComfyUI, Civitai, SDXL, Flux, ElevenLabs, Suno, Udio, self-hosted audio, and 3D generation, and notes that no elevenlabs SDK is pinned (the adapter is a direct HTTP client).

Related#

• Isis Generation Control
• Generation Audience Tiers and Surface Boundaries
• Creator Surfaces, Voice, Music, and 3D Generation
• Output Gallery, Lineage, Branch, and Replay
• Sophia Grounding
• Review, Compliance, and Trust & Safety
• Agent Registry, Job Orchestration, and Multi-Agent Plans
• Subsystem Glossary
• Architecture, Platform Foundations, and Security
• Hub: ../features.md