# Isis — Feature Reference > **Isis** is the Generative AI Factory of the Oshun platform. It handles all > AI-powered content generation — images, video, audio, 3D models, and text — > through a unified job submission system with GPU worker management, workflow > versioning, output tracking with full lineage and provenance, and quality > assurance. Isis provides the generation backbone that Yemaya, Hathor, Bellona, > and Lilith rely on for their creative production pipelines. Isis exposes three REST API services (Generation, Workflow, Output), a GPU worker process, a CLI, a web front-end, and a TypeScript client SDK, backed by 55 specialist libraries in `libs/isis/`. Every generation job is tracked with complete provenance — which provider, model, parameters, and hardware produced each output — enabling reproducibility, compliance, and cost accountability across the entire ecosystem. --- ## Platform Overview Isis operates as a generation-as-a-service platform internal to the Oshun ecosystem. Any domain — Yemaya for creative production, Hathor for worldbuilding assets, Bellona for engine-ready content, Lilith for meditation visuals — submits generation jobs through a REST API, CLI, or TypeScript SDK. Jobs are queued, routed to appropriate GPU workers, executed against the best available AI provider, and the results are stored with complete provenance and lineage tracking. ### Key Numbers The table below provides a quick orientation to the scope of the domain before diving into individual features. | Metric | Value | | ----------------------- | ------------------------------------- | | Generation Types | 15 modes across 5 media categories | | 3D Generation Providers | 7 cloud providers (+ local inference) | | Database Models | 24 (1141-line Prisma schema) | | API Services | 3 REST (Generation, Workflow, Output) | | Applications | 6 (3 APIs, GPU worker, CLI, web) | | GPU Worker Executors | 4 built-in executor types | | Asset-Pack Recipes | 32 typed workflow recipes | | CLI Command Groups | 7 | | Specialist Libraries | 55 directories (54 TS + 1 Python) | ### Core Capability Areas At the highest level, Isis delivers seven distinct capabilities. Each is described in detail in the sections that follow. | Capability | What Isis Provides | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Generation** | 15 modes: text-to-image, image-to-image, upscaling, inpainting, text-to-video, image-to-video, text-to-3D, image-to-3D, Gaussian splatting, mesh processing, texture upscale, Blender render, text-to-audio, voice synthesis, music generation | | **Provider Abstraction** | Unified adapter layer across LLM, image, video, video-processing, TTS, 3D, and ComfyUI providers; automatic routing and fallback | | **Workflow System** | Versioned, templated, composable multi-step generation workflows | | **Output Tracking** | Full lineage graph with provenance for every generated file | | **Quality Assurance** | Automated quality scoring, benchmarking, regression detection | | **Cost Management** | Budget enforcement, cost tracking, optimization routing | | **GPU Management** | Distributed GPU workers with multi-executor dispatching and VRAM estimation | --- ## Generation Capabilities ### 15 Generation Modes Isis supports 15 distinct generation types organized across five media categories. The mode determines which GPU executors and AI providers are eligible, which parameters are required, and what the output format will be. #### Image Generation | Mode | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TEXT_TO_IMAGE` | Generate images from text prompts with full control over model, dimensions, step count, guidance scale (how closely to follow the prompt), sampler algorithm, and random seed for reproducibility | | `IMAGE_TO_IMAGE` | Transform existing images using text prompts with configurable denoising strength — low strength preserves the original, high strength allows radical transformation | | `UPSCALING` | AI-powered image upscaling using ESRGAN, Real-ESRGAN, and BSRGAN algorithms (neural network models that reconstruct plausible high-frequency detail rather than just interpolating pixels) | | `INPAINTING` | Fill in or modify selected pixel regions of an existing image using a binary mask; surrounding context is preserved seamlessly | #### Video Generation | Mode | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `TEXT_TO_VIDEO` | Generate video clips from text descriptions via Zeroscope (open-source) or Runway (commercial); control duration, frame rate, and resolution | | `IMAGE_TO_VIDEO` | Animate a static image into a short video clip with configurable motion intensity and duration | #### 3D Generation | Mode | Description | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TEXT_TO_3D` | Generate complete 3D models from text descriptions using multi-provider routing to the best-fit specialist | | `IMAGE_TO_3D` | Reconstruct a 3D model from one or more reference photographs; quality improves with more distinct-angle reference images | | `GAUSSIAN_SPLATTING` | NeRF-based 3D scene reconstruction from unstructured photo collections (Gaussian splatting is a rendering technique where a 3D scene is represented as millions of tiny colored ellipsoids rather than polygons) | | `MESH_PROCESSING` | Post-processing and optimization of existing 3D meshes: decimation, smoothing, subdivision, UV unwrapping, LOD generation | | `TEXTURE_UPSCALE` | AI upscaling of existing 3D model textures to higher resolution using ESRGAN-family models | | `BLENDER_RENDER` | Submit Blender scene files to GPU workers for CPU/GPU rendering; supports Cycles and EEVEE render engines | #### Audio Generation | Mode | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | `TEXT_TO_AUDIO` | Generate sound effects, ambient soundscapes, and foley audio from text descriptions | | `VOICE_SYNTHESIS` | Text-to-speech with multiple voice options via ElevenLabs; supports emotion markers, pronunciation corrections, and voice cloning | | `MUSIC_GENERATION` | AI-composed music tracks with configurable mood, genre, tempo, key, and instrumentation | ### Job Lifecycle Every generation request follows a six-stage flow from initial submission to final delivery: 1. **Submit** — POST job with type, parameters, model selection, and priority level 2. **Queue** — Job enters a Redis-backed priority queue; position is queryable 3. **Route** — Job routed to an appropriate GPU worker based on `supportedTypes[]` and `supportedEngines[]` capabilities registered by each worker 4. **Execute** — Worker processes the job using the specified AI provider and parameters; progress percentage (0–100) reported during long-running jobs 5. **Store** — Results stored in S3-compatible object storage with full metadata, manifest, and provenance record 6. **Notify** — Optional webhook HTTP callback on completion or failure with full output information ### Job Management Once submitted, jobs can be monitored, cancelled, and replayed. The status field tracks the job through its complete lifecycle. - **Job Listing** — Browse all jobs with pagination and filtering by status, type, date range, and user - **Real-Time Status** — Job progresses through `PENDING → QUEUED → RUNNING → COMPLETED | FAILED | CANCELLED`; status queryable at any point - **Progress Tracking** — Percentage progress (0–100) reported during generation so UI can show meaningful progress indicators - **Job Cancellation** — Cancel pending, queued, or running jobs; resources are released immediately - **Queue Position Query** — Check the current queue position for waiting jobs to set user expectations - **Output Retrieval** — Download completed job outputs and full artifact manifests - **Webhook Notifications** — Configure per-job or per-workflow HTTP callbacks that fire on completion or failure - **Dead-Letter Queue** — Failed jobs land in a dead-letter queue for inspection and manual or automatic replay ### Priority Levels Jobs support four priority levels enforced at the Redis queue level. Higher priority jobs jump ahead of lower-priority ones in the queue; `URGENT` can interrupt running lower-priority work. | Priority | Behavior | | -------- | ---------------------------------------------------------------------- | | `LOW` | Processed after all higher-priority jobs; may be preempted by `URGENT` | | `NORMAL` | Standard FIFO processing within the normal queue | | `HIGH` | Processed ahead of LOW and NORMAL | | `URGENT` | Immediate processing; may interrupt lower-priority running jobs | --- ## Image Generation ### Stable Diffusion Ecosystem Isis's image generation is built on the Stable Diffusion model family. The following capabilities apply to both `TEXT_TO_IMAGE` and `IMAGE_TO_IMAGE` jobs. - **Model Selection** — Choose from any registered checkpoint in the model registry, including SD 1.5, SDXL, and community fine-tuned variants - **LoRA Support** — Apply one or more LoRA adapters (Low-Rank Adaptation — a lightweight fine-tuning method that adds a small set of trainable parameters to an existing model to shift its style or subject focus without full retraining) with configurable per-LoRA weight from 0 to 1 - **ControlNet** — Guided generation using structural conditioning signals: edge maps (Canny), depth maps, human pose skeletons (OpenPose), segmentation maps, line art, and normal maps - **Sampler Selection** — Euler, Euler Ancestral, DPM++, DPM++ SDE, DDIM, PLMS, and other samplers with configurable step count - **Negative Prompts** — Fine-grained negative prompt control to suppress unwanted content, artifacts, or styles - **Batch Generation** — Generate multiple variations from a single job submission with configurable batch size ### Advanced Techniques Several advanced conditioning techniques extend the base generation capabilities for specialized production needs. #### IP-Adapter IP-Adapter allows an image to be used as a conditioning signal alongside (or instead of) a text prompt, enabling visual consistency workflows that do not require fine-tuning. - **Image Prompting** — Use a reference image as a style or composition prompt in addition to or instead of text; useful for maintaining visual consistency across a series - **Style Transfer** — Transfer the artistic style of one image onto new content generated from text prompts - **Face Preservation** — Maintain facial identity across generated images without requiring LoRA training; useful for character consistency in production #### InstantID InstantID preserves a specific person's facial identity in generated images using a single reference photograph, with no fine-tuning required. - **Zero-Shot Identity Preservation** — Preserve a specific person's facial identity in generated images without any fine-tuning; requires only a reference photograph #### Florence-2 Florence-2 is a vision-language model used to extract structured captions from existing images, enabling "regenerate with variations" workflows. - **Vision-Language Understanding** — Extract structured semantic descriptions from existing images for use as generation prompts; enables "re-generate with variations" workflows from existing assets #### AnimateDiff Lightning AnimateDiff Lightning converts still images into short looping animations, useful for animated concept art and thumbnail creation. - **Animation from Still Images** — Create short looping animations from static images using AnimateDiff Lightning models; useful for animated concept art and thumbnail creation ### Inpainting Inpainting allows targeted modification of specific regions within an existing image, leaving the surrounding content untouched. - **Mask-Guided Editing** — Define pixel-precise binary masks for targeted region modification; regions outside the mask are not modified - **Seamless Blending** — AI blends the inpainted region with the surrounding image context for invisible edits - **Iterative Refinement** — Chain multiple inpainting passes on different regions for complex multi-region editing ### Live Preview - **Real-Time Generation Preview** — See image generation progress in real time as individual denoising steps complete; allows early cancellation if generation is going wrong --- ## Video Generation and Processing ### AI Video Generation (`@isis/ai-video`) Isis supports two primary video generation providers and a rich set of post-processing capabilities for enhancing and transforming generated video. - **Zeroscope Integration** — Text-to-video generation via the Zeroscope open-source model; parameters include duration, frame rate, resolution, and guidance scale - **Runway Integration** — Professional video generation via Runway Gen-3 API; higher quality at commercial cost with support for motion brush - **Keyframe Control** — Define specific keyframe images to guide video generation through predefined visual waypoints - **Image-to-Video** — Animate a static reference image into a video with configurable motion and duration ### Video Post-Processing Isis provides a comprehensive GPU-accelerated video enhancement pipeline via `@isis/video-enhancement`. The pipeline is organized into three processing tiers — NVIDIA hardware-accelerated, RIFE frame interpolation, and Topaz AI enhancement — which can be combined into pre-built workflows. #### NVIDIA RTX Video Processing NVIDIA's neural video processing runs directly on RTX hardware for the best throughput. - **Video Super Resolution (VSR)** — AI upscaling of video using NVIDIA RTX neural networks; four quality modes (fast, balanced, quality, ultra quality) with scale factors of 1.5×, 2×, and 4× - **HDR Conversion** — Convert standard dynamic range (SDR) video to high dynamic range (HDR); supports HDR10, HDR10+, Dolby Vision, and HLG output formats - **Color Space Conversion** — Transform between broadcast and cinema color spaces: BT.709, BT.2020, DCI-P3, sRGB #### RIFE Frame Interpolation RIFE (Real-Time Intermediate Flow Estimation) inserts AI-generated intermediate frames to increase effective frame rate without re-shooting. - **Frame Rate Enhancement** — Insert interpolated intermediate frames to double effective frame rate (e.g., 24fps → 48fps); reduces motion judder for playback - **RIFE Model Variants** — Supports RIFE versions 4.0 through 4.22, each with different speed/quality tradeoffs for different hardware budgets - **4K Support** — Frame interpolation at up to 4K resolution #### Topaz Video Enhancement Topaz's specialized AI models offer the highest quality enhancement at the cost of longer processing time. - **AI Upscaling** — Video upscaling using Topaz specialized models: Proteus (detail recovery), Artemis (standard), Gaia (highest quality), Chronos (motion), Iris (faces), Nyx (low light) - **AI Denoising** — Temporal-aware video denoising that maintains consistency across frames - **Video Stabilization** — Remove unwanted camera shake and micro-jitter with configurable stabilization strength - **Topaz Frame Interpolation** — Topaz-powered frame rate doubling as an alternative to RIFE #### Pre-Built Video Processing Workflows Six pre-built workflows combine the individual enhancement stages for common delivery scenarios: - **4K Upscale Workflow** — Combines VSR upscaling and Topaz enhancement for 4K delivery output - **Frame Interpolation Workflow** — Doubles frame rate with motion blur handling - **HDR Conversion Workflow** — SDR-to-HDR full pipeline with color space management - **Restoration Workflow** — Video restoration for archival and historical footage - **Streaming Workflow** — Optimized processing for streaming platform delivery - **Professional Workflow** — Full production pipeline applying all enhancement stages in the correct order ### Video-to-Mesh (`@isis/video-to-mesh`) - **Video-to-3D Geometry** — Extract 3D geometry from video footage using monocular depth estimation and multi-frame reconstruction; creates approximate 3D scenes from existing video without any 3D scanning equipment --- ## 3D Generation Pipeline The 3D generation pipeline is the most complex part of Isis. It spans seven cloud providers for initial mesh generation, a rich post-processing pipeline to bring meshes to production topology standards, automated rigging, texture enhancement, Gaussian splatting for photo-based reconstruction, and a sophisticated scene composition system. Sixteen libraries in `libs/isis/` implement this pipeline end-to-end. ### Multi-Provider 3D Generation (`@isis/3d-generation`) `@isis/3d-generation` integrates seven cloud 3D generation providers (plus a `local` inference target) with automatic routing to the best-fit provider based on input type and quality requirements. Each provider has its own adapter under `src/providers/`: | Provider | Adapter | Role | | -------- | ------------------ | ---------------------------------------------- | | Rodin | `RodinProvider` | Character and complex-object generation | | Meshy | `MeshyProvider` | Text-to-3D and image-to-3D with auto-texturing | | Tripo | `TripoProvider` | Single-image 3D reconstruction | | Trellis | `TrellisProvider` | High-geometric-detail 3D generation | | Hunyuan | `HunyuanProvider` | Large-scale 3D model generation | | ThreeDFY | `ThreeDFYProvider` | Production-quality 3D asset generation | | Marble | `MarbleProvider` | World Labs Marble image-to-splat environments | The `Generation3DProvider` type union enumerates `rodin`, `meshy`, `tripo`, `hunyuan`, `trellis`, `threedfy`, `marble`, and `local`. ### 3D Model Post-Processing (`@isis/3d-post-pipeline`) Raw meshes from generation providers rarely meet production topology standards. The post-processing pipeline applies a series of operations to prepare meshes for game engines, film pipelines, or print: - **Mesh Optimization (Decimation)** — Reduce polygon count while preserving visual silhouette and surface detail; configurable target polygon count or reduction ratio - **Interior Face Removal** — Detect and remove geometry hidden inside the model that is never visible to cameras; saves GPU memory and rendering time - **Mesh Smoothing** — Apply Laplacian or Taubin smoothing algorithms with configurable iteration count and smoothing factor - **Subdivision** — Increase mesh resolution via Catmull-Clark or Loop subdivision with configurable subdivision levels; useful before baking hi-poly detail - **UV Unwrapping** — Automatic UV map generation that minimizes seam visibility and maximizes texel density across the surface - **Texture Baking** — Project high-poly mesh surface detail (normals, ambient occlusion, curvature) onto a low-poly mesh with UV map for game-ready assets - **LOD Generation** — Automatically generate Level of Detail (LOD) meshes at configurable quality levels (LOD0=original, LOD1=50%, LOD2=25%, LOD3=10%) for runtime performance optimization in game engines - **Mesh Statistics** — Report vertex count, face count, edge count, non-manifold edges, degenerate faces, and topology quality score ### Topology Verification (`@isis/3d-quality-gates`) Before a 3D asset enters the production pipeline, `@isis/3d-quality-gates` validates its topology against configurable preset standards. This catches problems that would only manifest later during subdivision, deformation, or engine import. - **Quad-Mesh Analysis** — Measure the ratio of quad, triangle, and n-gon (polygon with more than 4 sides — generally undesirable in production) faces; n-gons cause subdivision and deformation artifacts - **Pole Detection** — Identify topology poles (vertices with more or fewer than 4 connected edges) that can cause pinching during subdivision or deformation - **Edge Flow Metrics** — Measure how well edge loops follow anatomically correct flow patterns for humanoid and organic models - **Topology Presets** — Pre-configured acceptance standards for game real-time (tight polygon budget, no n-gons), film (subdivision-ready, all quads), CAD (exact geometry priority), and 3D printing (manifold, no holes) ### Auto-Rigging (`@isis/universal-rigging`) Automated skeletal rigging eliminates the need for manual bone placement on common character types. A new engineer should understand that rigging is the process of creating a skeleton (hierarchy of bones) inside a 3D model and assigning mesh vertices to those bones — this is what makes a character move. Isis automates the parts that are normally done by hand. - **Skeleton Templates** — Humanoid, quadruped, bird, fish, insect, serpent, and custom skeleton configurations; each with species-appropriate joint hierarchy - **Bone Naming Conventions** — Mixamo, Unreal Engine, Unity (Mecanim), Blender, and fully custom naming standards; ensures compatibility with motion capture retargeting and animation systems in target engines - **Facial Rig Configuration** — Automated facial bone placement with configurable blend shape count and joint density; supports both joint-based and shape-key facial animation workflows - **Control Rig Generation** — IK/FK (Inverse Kinematics/Forward Kinematics — two complementary methods for controlling character limb poses) control rig with pole vector targets and joint constraints for animator-friendly setup - **Skin Weight Algorithms** — Heat map (fast, physically-plausible), geodesic (follows surface distance), voxelized (handles complex topology), and nearest-bone (simple, fast) skinning weight calculation methods ### Texture Enhancement (`@isis/ai-texturing`) Advanced AI texture processing for generated and existing 3D assets: - **Texture Upscaling** — AI upscaling using ESRGAN, Real-ESRGAN, and BSRGAN models; recovers plausible fine detail lost in original low-resolution textures - **Seamless Tiling** — Convert any texture to seamlessly tile without visible repeating seams; essential for terrain and fabric materials - **Material ID Extraction** — Automatically detect and separate distinct material regions from a combined texture atlas into individual material maps - **UDIM Layout Support** — Handle UDIM tile layouts (a production standard where a single mesh uses multiple texture tiles for very high resolution) used in film and premium game productions - **Batch Processing** — Process dozens of textures in parallel with configurable concurrency ### Gaussian Splatting (`@isis/gaussian-splatting`) Gaussian splatting is a technique for creating photorealistic 3D scene representations from ordinary photographs. Unlike polygon meshes, a Gaussian splat represents the scene as millions of tiny colored ellipsoids that are rendered directly. `@isis/gaussian-splatting` handles the full pipeline from photo collection to final splat or polygon mesh. - **Scene Reconstruction** — Create a 3D Gaussian splat scene representation from an unstructured collection of photographs; captures real-world lighting and appearance with photo-realistic fidelity - **Point Cloud Processing** — Filter and refine the sparse point cloud produced as an intermediate step in reconstruction - **Mesh Extraction** — Convert Gaussian splat results to traditional polygon meshes using marching cubes or Poisson reconstruction; enables use in polygon- based game engines - **Splat Rendering** — Native Gaussian splat rendering for direct in-engine use where splat renderers are available ### Scene Composition from Image (`@isis/scene-from-image-composer`) The scene composer is the most sophisticated module in the 3D pipeline. It takes a single input image and produces a complete 3D scene — environment, foreground objects with proper meshes, lighting, and ambient audio — through a multi-stage pipeline that coordinates environment generation, object detection, 3D model generation, mesh-to-Gaussian baking, lighting estimation, and persistence. It implements all nine ISIS_GAPS plan blocks. - **Environment Generation (World Labs Marble)** — Image-to-splat scene with bounds, preview URLs, and ambient prompt extraction - **Object Source Modes** — Discriminated union over explicit-prompt / caller-cropped / VLM-detected / mixed; auto-detection via grounding adapter with IoU merge, foreground classifier, topK and cost-cap clamps - **Mesh Provider Router** — Weighted scoring across Hunyuan / Tripo / Meshy with quality, cost, and latency weights plus per-spec capability matching - **Anchor Constraint Solver** — Anchor DSL with `centre-ground`, `on-floor-front-centre`, `bbox-floor`, `forward-of`, `right-of` relations; single-batch solver produces per-mesh 4x4 column-major transforms - **Mesh-to-Gaussians Bake** — Surface-sampling baseline (Poisson area-weighted oriented disks) plus differentiable hybrid baker (CPU rasteriser warm start plus gradient-descent refinement of positions, colours, opacities) plus splat merging (concat, depth-sort, occlusion-cull, dedupe, bounds recompute) - **Lighting Consistency** — SH9 environment-light estimation from the Marble splat, applied to mesh PBR before sampling; composition coherence gate with pluggable feature adapter - **Bit-Exact Niantic `.spz` Codec** — Encoder and decoder validated via in-process round-trip tests against the Niantic reference format - **Job-Envelope Dispatch** — Pluggable JobChannel transport (Redis for production, in-memory for tests); heavy phases (env, objects, bake) route to a GPU worker pool with subscribe-before-publish ordering and per-jobId response channels - **Postgres Persistence and SIGKILL-Resume** — Six Prisma models, in-memory and Postgres repository implementations, SIGKILL-resume golden asserting manifest equivalence after mid-run crash - **Policy Pipeline** — Six external isis libraries threaded through the runner: gallery registration, lineage builder, token-budget enforcer, model-governance-3d license compliance, anomaly-detection drift watch, agent-consensus checkpoint handler - **Convenience and Advanced APIs** — `composeSceneFromImage()` for one-shot use, `createSceneFromImagePipeline()` for caller-supplied orchestrators with full block-A event stream ### Prompt Engineering (`@isis/prompt-engineering`) Good prompts are essential for 3D generation quality. `@isis/prompt-engineering` provides tooling to build, analyze, and improve generation prompts across all asset types. - **Built-In Prompt Templates** — 10 category-specific templates optimized for each asset type: humanoid character, creature, weapon, prop, environment, vehicle, architecture, furniture, food, plant - **Interactive Prompt Builder** — Visual builder that assembles a prompt from selected style modifiers, material descriptors, lighting conditions, and level-of-detail specifications - **Prompt Analyzer** — Analyzes a given prompt for common failure patterns (conflicting descriptors, underspecified detail, known model weaknesses) with severity ratings and corrective suggestions - **AI Prompt Optimizer** — Rewrites prompts using patterns from successful historical generations to improve expected output quality - **Prompt History** — Searchable history of all submitted prompts with quality ratings and usage counts for building institutional prompt knowledge ### Reference Image Processing When generating 3D models from images, the quality of the input reference photographs has a large impact on reconstruction quality. These tools prepare reference images for optimal results. - **Background Removal** — Automatic subject extraction from reference photographs; clean subject isolation significantly improves image-to-3D reconstruction quality - **Multi-View Generation** — Generate a complete 6-view or 8-view reference set (front, back, left, right, top, bottom, and diagonals) from a single image using a specialized multi-view diffusion model - **Image Preprocessing** — Resize, sharpen, normalize exposure, and adjust white balance for optimal generation input quality ### 3D Asset Library and Browser (`@isis/3d-asset-library`, `@isis/3d-browser`) Generated 3D assets can be previewed directly in the browser and exported in multiple engine-native formats without downloading files first. - **In-Browser 3D Preview** — Real-time WebGL preview of generated 3D models directly in the browser without downloading - **Orbit/Pan/Zoom Controls** — Standard orbit camera controls for inspecting generated models from all angles - **Material Preview** — Preview textures, materials, and shading on the model in the browser viewer - **Format Export** — Export generated assets in engine-native formats (GLTF, FBX, OBJ, USDZ, Blender native) ### 3D Benchmarking (`@isis/3d-generation-benchmarks`) With seven cloud providers available, operators need objective comparisons to decide which providers to use for which tasks and to detect provider quality regressions. - **Cross-Provider Benchmarks** — Submit identical prompts to all configured 3D providers simultaneously and compare visual quality, generation time, and cost - **Throughput Benchmarks** — Measure concurrent generation capacity across hardware configurations - **Regression Testing** — Detect quality regressions when provider models update by comparing against historical benchmark baselines ### Local Inference (`@isis/3d-inference-local`) - **Local GPU Execution** — Run supported 3D generation models directly on local GPU hardware, eliminating cloud provider dependency and API costs for high-volume workflows ### 3D Semantic Editing (`@isis/3d-semantic-editing`) After a 3D model is generated, semantic editing allows targeted modifications using natural language rather than direct mesh manipulation. - **Text-Guided 3D Modification** — Modify specific attributes of generated 3D models using natural language instructions (e.g., "make the cape longer", "add more detail to the face") - **Part Selection** — Select and independently modify specific named parts of a model using segmentation-aware editing - **Style Transfer on 3D** — Apply visual style textures to 3D model surfaces while preserving underlying geometry ### 3D Model Governance (`@isis/model-governance-3d`) AI generation models carry licensing terms that restrict commercial use. `@isis/model-governance-3d` enforces those terms automatically. - **License Tracking** — Record and enforce licensing terms for each 3D generation model; distinguishes commercial-licensed, personal-only, and open-source models - **Compliance Checks** — Verify that generation requests comply with model licensing terms and organizational compliance policies before execution - **Usage Attribution** — Maintain auditable records of which model produced each output for provenance and compliance reporting --- ## Audio Generation ### Audio Pipeline (`@isis/audio-generation`) Isis handles the full spectrum of audio generation needs: sound effects and ambient audio from text, high-quality voice synthesis, voice cloning, music composition, and conversational AI voice agents. - **Text-to-Audio** — Generate sound effects, ambient soundscapes, and foley audio from text descriptions; controls for duration, sample rate, and audio category - **ElevenLabs Voice Synthesis** — High-quality TTS using ElevenLabs with a library of pre-built voices, custom voice cloning from audio samples, and fine-grained emotion and delivery control via text markers - **Voice Cloning** — Clone an existing voice from a reference audio sample to create consistent narration across a production without re-recording - **Music Generation** — AI-composed original music tracks with configurable mood (calm, energetic, dark, joyful), genre (orchestral, electronic, acoustic), tempo (BPM), key signature, and instrumentation emphasis - **Conversational AI Agents** — ElevenLabs Agents Platform 2.0 integration for deploying multi-agent conversational AI with voice; configurable LLM backend (GPT-4, Claude, etc.) behind distinct voice personas --- ## Workflow System ### Workflow Definitions Workflows define reusable, composable generation pipelines that chain multiple generation steps with explicit dependencies. They are useful when a generation task always requires the same sequence of operations — for example, generating a character concept image, upscaling it, and then generating a 3D model from it. - **Workflow Creation** — Define multi-step workflows with step inputs, outputs, and dependency declarations; steps run in dependency order with optional parallelism - **Workflow Engines** — `COMFYUI`, `BLENDER`, `UNREAL`, `GODOT`, `CUSTOM` - **Workflow Categories** — `IMAGE_GENERATION`, `VIDEO_GENERATION`, `AUDIO_GENERATION`, `GENERATION_3D`, `UPSCALING`, `STYLE_TRANSFER`, `INPAINTING`, `COMPOSITING`, `RENDERING`, `SIMULATION`, `UTILITY`, `OTHER` - **Workflow Status** — `DRAFT`, `PUBLISHED`, `DEPRECATED`, `ARCHIVED` - **Visibility Controls** — `PRIVATE`, `TEAM`, `ORGANIZATION`, `PUBLIC` - **Run Statistics** — Tracks total runs, successful runs, and average execution time per workflow for performance monitoring ### Workflow Versioning Workflow versioning works like Git for generation pipelines: every change creates a new immutable snapshot, and the history is always available for rollback or comparison. - **Immutable Versions** — Every change to a workflow creates a new immutable version with a complete configuration snapshot; prior versions are never modified - **Changelog Tracking** — Human-readable changelog between versions captured at version creation time - **Version Comparison** — Diff any two versions of a workflow to see exactly what changed - **Version Rollback** — Revert a workflow to any previous version with a single API call - **Deprecation Marking** — Mark outdated versions as deprecated while preserving them in the version history for audit purposes ### Workflow Templates Templates are curated starter workflows that users can customize for their own purposes. They lower the barrier to entry for new platform users. - **Template Library** — Curated starter templates for common patterns (text-to- image with upscale, image-to-3D with rigging, TTS pipeline) with usage statistics - **Template Variables** — Parameterized templates with typed inputs, default values, and validation schemas; fill in variables at execution time - **Featured Templates** — Curated high-quality templates promoted in the template browser for discoverability - **Template Starring** — Bookmark frequently-used templates for quick access ### ComfyUI Integration (`@isis/comfyui-sdk`, `@isis/comfyui-nodes`) ComfyUI is a node-based visual workflow editor for Stable Diffusion and related models. Isis integrates with it both programmatically (via the SDK) and via custom nodes that expose Oshun-specific capabilities inside the ComfyUI graph. - **ComfyUI SDK** — WebSocket-based SDK for executing ComfyUI workflows programmatically from TypeScript code without a browser interface - **Custom Nodes** — Oshun-specific ComfyUI custom nodes that expose Isis capabilities (provider routing, lineage tracking, quality gates) as graph nodes in ComfyUI - **Workflow Import** — Import existing ComfyUI workflow JSON files directly into the Isis workflow registry; any ComfyUI workflow becomes an Isis workflow - **RunComfy Integration** — Serverless ComfyUI execution via RunComfy; submit workflows to cloud GPU infrastructure without managing local ComfyUI instances --- ## AI Provider Ecosystem ### `@isis/ai-providers` — Unified Provider Adapter A unified interface abstracts all provider-specific APIs behind a common contract. Provider selection can be fully automatic (based on job type and routing strategy) or manually specified per job. The `providers/` tree contains adapter sets for the following categories (`libs/isis/ai-providers/src/providers/`): | Category | Providers | | -------------------- | ------------------------------------------------------------------------------- | | **LLM / Text** | Anthropic (Claude), OpenAI (GPT), Google (Gemini), xAI (Grok), Ollama (local) | | **Image** | Stability AI (SD / SD3.5), Black Forest Labs (FLUX) | | **Video** | Hunyuan, LTX, Wan | | **Video Processing** | NVIDIA RTX, RIFE, Topaz | | **Voice / TTS** | ElevenLabs | | **3D** | ComfyUI-routed 3D (Hunyuan-3D, Trellis), Gaussian-splat CUDA, Blender auto-rig | | **ComfyUI** | Local ComfyUI and ComfyUI Cloud | | **Civitai** | Civitai model intake, Civitai Link, content safety, LoRA training, monetization | Provider routing (`orchestration/`), provenance, release gates, and a factory (`factory/`) complete the package. (Multi-provider 3D generation across Rodin, Meshy, Tripo, Trellis, Hunyuan, ThreeDFY, and Marble is provided by the separate `@isis/3d-generation` library — see the 3D Generation Pipeline section.) ### `@isis/llm-providers` and `@isis/llm-orchestrator` Additional LLM provider abstractions and multi-step orchestration for workflows that combine generation with text analysis, prompt refinement, or quality assessment using language models. ### `@isis/batch-llm-processing` Batch processing utilities for high-volume LLM inference: request batching, rate limit management, cost tracking per batch, and result collation. ### Provider Factory The provider factory manages provider instance lifecycle to avoid repeated authentication round-trips. - **Auto-Instantiation** — Create provider instances from configuration objects with a single factory call; no manual SDK initialization per provider - **Instance Caching** — Reuse initialized provider instances across requests to avoid repeated authentication overhead - **Typed Error Handling** — Each provider category surfaces a typed error class with a category-specific error-code enum (`LLMError`, `VideoProcessingError`, `CivitaiError`, `ComfyCloudError`, `MultiGPUError`, `ModelMergingError`, and others) so callers can branch on failure cause ### Intelligent Routing The Unified Generation Pipeline selects the best provider for each request based on four configurable strategies. - **Capability Matching** — Automatically match request requirements (generation type, format, resolution) to providers that support them - **Quality Tier Selection** — Route to different provider tiers based on the requested quality (draft → cheap/fast, production → highest quality) - **Fallback Chains** — When a provider fails or is unavailable, automatically retry with the next-best provider in the configured fallback chain - **Routing Strategies** — Cost-optimized (minimize spend), quality-optimized (maximize output quality), latency-optimized (minimize time to result), and balanced (optimize a weighted combination) --- ## GPU Worker Management ### Worker Architecture The GPU worker (`apps/isis/gpu-worker`) is the sole consumer of the Redis job queue. It operates on a poll model — no jobs are pushed to it — which provides clean at-least-once delivery semantics even across worker restarts. 1. Consumes jobs from a Redis-backed queue (BullMQ) 2. Routes each job to a specialized executor based on generation type 3. Reports progress percentage (0–100) during long-running jobs 4. Emits structured in-process lifecycle events: `worker:started`, `worker:stopped`, `worker:health`, `job:started`, `job:progress`, `job:completed`, `job:failed` 5. Stops gracefully — `stop()` closes the queue worker, which waits for the current job to complete before shutting down executors and connections ### Four Built-In Executors Specialized executors handle the most compute-intensive job types directly on the GPU, while all other types route through the generic executor that calls external AI provider APIs. | Executor | Generation Types | Description | | ----------------------------- | -------------------- | ---------------------------------------- | | `texture-upscale-executor` | `TEXTURE_UPSCALE` | AI texture upscaling via ESRGAN variants | | `blender-render-executor` | `BLENDER_RENDER` | Blender Cycles/EEVEE scene rendering | | `gaussian-splatting-executor` | `GAUSSIAN_SPLATTING` | NeRF/Gaussian splatting reconstruction | | `mesh-processing-executor` | `MESH_PROCESSING` | 3D mesh optimization and post-processing | ### Worker Configuration Each worker instance registers itself in the `GpuWorker` database table with its capabilities and hardware details. The generation API consults these registrations when routing jobs. - **Worker ID** — Unique identifier for each worker instance for tracking and debugging - **Worker Type** — `general` (handles all supported types) or `specialized` (single type, higher efficiency for that type) - **GPU Device Selection** — Configurable CUDA device indices via `GPU_DEVICES` environment variable; supports multi-GPU machines with separate workers per GPU - **Work Directory** — Temporary working directory for job processing; cleaned after each job - **Model Cache Directory** — Persistent cache for downloaded model checkpoints; avoids re-downloading models between jobs - **Concurrency** — Maximum number of simultaneous jobs per worker instance ### Multi-GPU Orchestration Multiple GPU workers run simultaneously, each registered in the `GpuWorker` database table. The generation API distributes jobs across available workers based on their declared `supportedTypes[]` and `supportedEngines[]` capabilities: - **VRAM Estimation** — Estimate VRAM requirements from job parameters before dispatching; prevents out-of-memory failures by routing to workers with sufficient GPU memory - **Hardware Detection** — Automatic detection of available GPU hardware, compute capability, and VRAM for capability registration - **Memory Optimization** — Configurable memory optimization modes for consumer (8-16GB VRAM), professional (24-48GB VRAM), and datacenter GPUs --- ## Output Management and Lineage ### Output Manifest (`@isis/outputs`) Every generated file is tracked in the `generated_outputs` table with a comprehensive manifest. This tracking is what enables storage lifecycle management, deduplication, and the lineage graph. - **File Identity** — Original filename, MIME type, file size, and SHA-256 hash for integrity verification and deduplication - **Storage Location** — Bucket name, storage key path, and direct access URL for retrieval - **Media Properties** — Width and height (images/video), depth (3D models), duration and frame count (video/audio) - **Quality Tier** — `@isis/outputs` classifies outputs into a `QualityTier` of `draft`, `production`, or `master` - **Storage Tier** — `HOT`, `WARM`, `COLD`, or `GLACIER` (see Storage Tiering) - **Access Tracking** — Access count and last-accessed timestamp for lifecycle management decisions ### Storage Tiering Generated files that are not accessed frequently automatically migrate to cheaper storage classes. The following tiers form a cost-vs-access-speed ladder from most accessible to least expensive: | Tier | Storage Class | Default Retention | | --------- | ------------------------------------------- | ----------------- | | `HOT` | Standard S3 — immediate access | 30 days | | `WARM` | Infrequent access storage class | 90 days | | `COLD` | Glacier or equivalent archive | 365 days | | `GLACIER` | Deep archive — retrieval latency acceptable | Configurable | ### Lineage Graph The lineage graph answers the question: "where did this file come from?" `LineageEdge` records link derived outputs back to their sources using typed relationship labels: | Edge Type | Description | | ---------------- | -------------------------------------------------- | | `DERIVED_FROM` | Output is a general derivative of a source | | `COMPOSED_OF` | Output is a composition of multiple source outputs | | `REFINED_FROM` | Output is a refined/improved version of a source | | `UPSCALED_FROM` | Output is an upscaled version of a source | | `CONVERTED_FROM` | Output is a format-converted version of a source | A `LineageEdge` also carries an optional `transformation` description and an optional `weight` (default `1.0`) for composed outputs. The following example shows a typical 3D asset production chain where an initial generation is refined and then converted to an engine-native format: ``` Original Prompt → TEXT_TO_3D (Meshy) → Raw Mesh │ REFINED_FROM ▼ Optimized Mesh (LOD0) │ CONVERTED_FROM ▼ Engine-Native .uasset ``` ### Provenance Records Every output has an associated `Provenance` record capturing the complete generation context. This record is what makes outputs reproducible — given a provenance record, you can re-run the same job and get an identical or equivalent output. - Workflow metadata and exact version used - Model information (checkpoint name, version, integrity hash) - All LoRA adapters applied with their weights - Full parameter set (seed, steps, CFG scale, sampler, scheduler, etc.) - GPU hardware type and total generation time - Provider used and estimated generation cost --- ## Quality Assurance ### 3D Quality Gates (`@isis/3d-quality-gates`) Quality gates run automatically before 3D assets enter the production pipeline. They catch topology problems and asset defects that are cheap to fix during generation but expensive to fix after an asset has been integrated into a game or film pipeline. - **Topology Validation** — Verify quad distribution ratios, detect problematic poles, validate edge flow quality - **UV Coverage Analysis** — Check UV map completeness (no missing UVs) and texel density efficiency (no wasted UV space) - **Texture Resolution Check** — Verify textures meet minimum resolution requirements for the target platform and quality tier - **Rig Validation** — Verify skinning weight completeness (every vertex weighted), weight normalization (weights sum to 1.0), and bone assignment validity - **LOD Chain Validation** — Verify that each LOD level is a proper simplification of the previous level within acceptable visual deviation bounds ### 3D Benchmarks (`@isis/3d-generation-benchmarks`) - **Cross-Provider Comparison** — Compare output quality, generation speed, and cost across all configured 3D providers for identical input prompts - **Throughput Measurement** — Measure concurrent generation capacity and per-request latency under various load conditions - **Regression Detection** — Automatically detect quality regressions when providers update their models by comparing new outputs against historical baselines ### User Feedback Loop The `@isis/ai-providers` quality-assurance pipeline models a `UserFeedback` record (rating, thumbs up/down, comparison winner, free-text annotations, `feedbackType`) per output and provider, and `aggregateFeedbackInsights()` rolls that feedback up to surface quality trends used for routing decisions. --- ## Cost Tracking and Optimization ### Per-Job Cost Tracking Every generation job records its cost at multiple points in the pipeline, enabling accurate attribution and historical analysis. - **Cost Estimate** — Every `GenerationJob` carries a `costEstimate` (USD), and text jobs additionally record `tokensUsed`; GPU jobs record `gpuTimeSeconds` and `gpuType` - **Actual Cost in Provenance** — The `Provenance` record stores the actual `cost` and the `provider` used, alongside the full parameter set and GPU - **Historical Backfill** — Admin routes (`POST /api/v1/jobs/admin/costs/backfill`) recompute missing historical costs from provider cost-log entries ### Routing-Driven Optimization Provider routing in `@isis/ai-providers` supports cost-optimized, quality-optimized, latency-optimized, and balanced strategies (see the [AI Provider Ecosystem](#ai-provider-ecosystem) section). The `@isis/operation-orchestrator` library additionally computes a batch-composition score to group requests for providers that support batch pricing. LLM-specific cost prediction, budgets, and enforcement are provided by `@isis/token-budget` — see [Token Budget Management](#token-budget-management). --- ## Model Registry and Discovery ### Internal Model Registry (`ModelRegistry` table) The model registry is the central catalog of all AI model checkpoints available to Isis workers. When a job specifies a model, the worker resolves it here to get the storage path, format, and metadata. - **Model Types** — `CHECKPOINT`, `LORA`, `CONTROLNET`, `VAE`, `EMBEDDING`, `UPSCALER`, `CUSTOM`; each type has type-specific metadata fields - **Format Support** — `SAFETENSORS`, `PYTORCH`, `ONNX`, `GGUF`, `BLEND`, `GLTF`, `FBX`, `OTHER` - **Model Metadata** — Version string, integrity hash (SHA-256), file size, trigger words (tokens that activate LoRA or embeddings), compatible engine list - **Visibility** — Public (available to all users) or private (scoped to a specific organization) - **Pre-Signed Upload** — Upload new models to S3 via pre-signed URLs without routing large files through the API server; confirm completion to register the model ### Civitai Integration Civitai is the largest community for sharing Stable Diffusion model checkpoints, LoRAs, and other assets. Isis integrates deeply with it for model discovery and download. - **Model Discovery** — Search and browse Civitai models by type, base model, architecture, category, and tag - **Model Download** — Download checkpoints, LoRAs, embeddings, VAEs, and ControlNet models directly from Civitai into the Isis model registry - **Civitai Link** — Connect local ComfyUI installations to Civitai for seamless model management and generation from local hardware - **Popularity Metrics** — Surface models by download count, generation count, and community rating ### LoRA Fine-Tuning (`@isis/model-fine-tuning`) LoRA (Low-Rank Adaptation) training allows operators to create custom model adapters that shift a base model's style or subject focus toward a specific look or character, using a small image dataset and far less compute than full fine-tuning. - **Custom LoRA Training** — Train LoRA adapters on user-provided image datasets for character consistency, style anchoring, or specific subject focus - **Dataset Management** — Prepare, caption, and manage training image datasets including automatic captioning using BLIP/Florence-2 - **Training Configuration** — Configure learning rate, training steps, batch size, and regularization parameters - **Training Progress Monitoring** — Real-time loss curve tracking and intermediate sample generation during training ### Model Comparison and Selection When multiple models are available for a task, these tools help operators choose the best one objectively. - **Standard Benchmark Prompts** — Run any set of models against the same standardized prompt suite for fair quality comparison - **Side-by-Side Comparison** — Compare generation outputs from multiple models across quality dimensions (sharpness, coherence, style fidelity, detail) - **Statistical Analysis** — Significance testing on quality score differences to identify genuinely better models versus noise - **AI Recommendation Engine** — Recommends the best model for a given use case based on stated requirements and historical performance data --- ## Security and Anomaly Detection `@isis/anomaly-detection` provides suspicious-activity detection, chargeback prediction, and account protection for the domain. Its service exposes `analyzeTransaction()`, `analyzeSession()`, and `predictChargeback()` over a configurable rule engine. ### Suspicious Activity Detection Because AI generation can be expensive, abusive usage patterns — bulk generation farming, credential sharing, API abuse — need to be detected and blocked automatically. - **Usage Pattern Analysis** — Continuously monitors generation frequency, API call patterns, and resource consumption per user for behavioral anomalies - **Activity Thresholds** — Configurable thresholds for generation frequency (requests per minute), burst activity, and unusual parameter patterns - **Risk Scoring** — Multi-factor risk level assessment that combines behavioral signals into a single LOW/MEDIUM/HIGH/CRITICAL risk score - **Automated Responses** — Configurable automated responses per risk level: alert only, throttle rate limits, restrict generation types, suspend account, or ban ### Chargeback Prevention - **Chargeback Prediction** — ML-based prediction of potential payment chargebacks based on transaction characteristics - **Real-Time Transaction Monitoring** — Monitor payment transactions for indicators of fraudulent use ### Account Protection - **Login Pattern Analysis** — Detect account compromise indicators from unusual login location, time, or device patterns - **Session Management** — Automatic session invalidation when suspicious account activity is detected; requires re-authentication ### Content Safety (Civitai integration) The `@isis/ai-providers` Civitai provider set includes a content-safety provider (`content-safety-provider.ts`) for classifying model and image content as part of the Civitai integration path. --- ## Job Envelope and Operation Orchestration ### Job Envelope Schema (`@isis/job-envelope`) The job envelope is the canonical message format for every generation job that flows through the Isis queue system. It is intentionally separate from the `GenerationJob` database record — the envelope is a compact, strict message designed for reliable queue serialization, while the database record is the full lifecycle-tracking store. Defining the envelope as a shared library ensures that all producers (API, SDK, CLI) and all consumers (GPU workers, executors) share an identical, versioned message contract. - **Versioned Schema** — The `ISIS_JOB_ENVELOPE_VERSION` constant ensures producers and consumers can detect version mismatches before attempting to process a message; old workers reject envelopes from newer producers gracefully - **Generation Type Manifest** — `ISIS_GENERATION_TYPE_MANIFEST` maps every generation type to its required parameters, optional parameters, and supported output formats; used by both validation and routing logic - **Control Input Definitions** — Canonical definitions for ControlNet modes, IP-Adapter modes, and InstantID modes including their input source kinds, blend domains, and preset IDs; ensures all services agree on parameter names and valid values - **Priority Schema** — Typed priority levels (LOW, NORMAL, HIGH, URGENT) with Zod validation for all queue submissions - **Type Aliases** — Backwards-compatible aliases for renamed generation types; prevents breaking changes when type names are standardized ### Operation Orchestration (`@isis/operation-orchestrator`) The operation orchestrator manages the lifecycle of complex multi-step generation operations that go beyond what a single job can express. It models retry policies, batching, dead-letter queues, and execution chains as first-class entities with their own queryable state. - **Retry Orchestration** — Manages retry attempts for failed operations with configurable retry policies (max attempts, backoff strategy, jitter) per failure classification; distinguishes transient failures from permanent ones and applies appropriate retry behavior to each - **Batch Composition** — Groups multiple generation requests into optimized batches for providers that support batching; computes a batch composition score to maximize cost efficiency while minimizing total wait time - **Dead Letter Management** — Failed operations that exhaust their retry policy enter the dead letter queue with a structured failure record; the dead letter service provides inspection, manual replay, and bulk resolution tooling - **Execution Chains** — Models ordered sequences of dependent operations as a first-class entity; chain status (phases, progress, blocked steps) is queryable at any point during execution - **Pipeline Execution Tracking** — Tracks multi-phase pipeline executions with per-phase start times, completion times, and output manifests; provides a complete execution timeline for debugging and performance analysis - **Failure Classification** — Classifies each failure into actionable categories (transient network, rate limit, provider error, invalid input, resource exhaustion, timeout) so retry logic and alerting can respond appropriately --- ## Token Budget Management ### LLM Cost Controls (`@isis/token-budget`) The token budget library provides fine-grained control over LLM API spending at every level of the organization. This is a critical capability when autonomous pipelines can generate thousands of LLM calls in a single run — without budget enforcement, a single runaway pipeline could exhaust a month's budget in hours. - **Budget Allocation** — Allocate token budgets per tenant, per category (e.g., image generation vs. text generation vs. quality evaluation), and per time period (hourly, daily, monthly); allocations compose hierarchically so individual category budgets cannot exceed the tenant total - **Cost Prediction** — Predict the cost of a generation request before execution using model pricing tables and usage trend analysis; gives autonomous pipelines the ability to check affordability before committing to expensive operations - **Enforcement Rules** — Configure per-tenant enforcement rules that define what happens when a budget threshold is crossed: warn, throttle, block, or redirect to a cheaper fallback model - **Budget Alerts** — Emit structured alerts at configurable threshold percentages (50%, 75%, 90%, 100% of budget consumed); alerts include remaining budget, projected exhaustion time, and top spending categories for immediate action - **Optimization Recommendations** — Analyse spending patterns to recommend allocation adjustments, model substitutions, and request batching strategies that would reduce cost for equivalent quality - **Fallback Decision Engine** — When the primary model exceeds budget, the fallback engine selects the best available cheaper model using a ranked fallback table; tracks fallback decisions for transparency and auditing - **Budget Snapshots** — Capture point-in-time budget snapshots for historical comparison and trend analysis; useful for month-over-month spend reviews - **Multi-Tenant Isolation** — Each tenant's budget state is fully isolated; a tenant exceeding their budget has no effect on other tenants' generation capacity --- ## Multi-Agent Consensus ### Agent Debate and Reasoning (`@isis/agent-consensus`) The agent consensus library enables multiple AI agents to debate a question, evaluate evidence, and converge on a well-reasoned conclusion. Within Isis, this is used for quality evaluation, prompt optimization, and adversarial validation of AI-generated content. Any other domain can also use it for decisions that benefit from structured multi-perspective reasoning rather than a single model's answer. - **Debate System** — Structure a debate with a central claim, multiple agents representing distinct positions (advocate, skeptic, neutral evaluator), and a moderator that enforces debate rules; the debate produces a structured argument graph with each position backed by cited evidence - **Claim and Argument Modeling** — Every claim has a confidence score, a supporting argument list, and a counter-argument list; argument validity is assessed using formal fallacy detection that identifies common logical errors (ad hominem, straw man, false dichotomy, appeal to authority, hasty generalization) - **Reasoning Chain Tracking** — Each agent's reasoning is recorded as an ordered chain of steps with the type of inference used at each step (deduction, induction, abduction, analogy); the chain is auditable and can be replayed - **Evidence Evaluation** — Agents cite evidence for every claim; the system evaluates evidence quality (source credibility, relevance, specificity) and weighs it accordingly when computing claim confidence - **Perspective Aggregation** — Aggregate reasoning from multiple agents into a single coherent synthesis; the `AggregatedReasoning` type captures areas of consensus, areas of remaining disagreement, and conflicting premises for human review - **Convergence Sessions** — Formal sessions where agents work toward a shared position through compromise proposal generation; the session records how much convergence was achieved and which points remain unresolved - **Quality Metrics** — Measure debate quality across reasoning coherence, evidence diversity, argument balance, and convergence degree; low-quality debates can trigger a second debate round with additional evidence --- ## ReAct Agent Framework ### ReAct-Pattern Agent Framework (`@isis/react-framework`) ReAct (Reasoning + Acting) is an agent pattern where an AI model alternates between reasoning steps and tool use actions — each step's reasoning informs what tool to call next, and each tool result informs the next reasoning step. `@isis/react-framework` provides the execution infrastructure for these tool-using agents within Isis, including lifecycle management, full trace recording, and tooling for debugging where agents make poor decisions. - **Thought-Action-Observation Loop** — Agents execute in a structured loop: generate a Thought (reasoning step explaining what to do next), select and execute an Action (a tool call), receive an Observation (the tool's result), and repeat until the task is complete or the step limit is reached - **Tool Registry** — Register typed tool definitions with names, descriptions, input parameter schemas (Zod-validated), output schemas, and timeout limits; the agent receives the tool list and selects tools by name based on its reasoning - **Action Alternatives** — For each action, the framework records alternative actions that were considered but not chosen; provides explainability and allows human reviewers to assess whether the agent made the best selection - **Agent Run Lifecycle** — Full lifecycle management for agent runs: start, step execution, pause, resume, cancellation, and completion with a complete trace record; runs can be inspected mid-execution for debugging - **Trace Recording** — Every agent run produces a structured trace (sequence of ReAct steps) that is stored for replay, audit, and performance analysis; traces enable identifying where agents get stuck or make poor decisions - **Step Limit Enforcement** — Configurable maximum step count per agent run prevents runaway agents from executing indefinitely; exceeded step limits produce a graceful failure with a partial trace rather than a hard crash --- ## CLI Tools The `isis` CLI (`apps/isis/cli`) provides seven command groups for complete platform interaction from the terminal. It is the recommended tool for scripted generation workflows, CI/CD integration, and batch operations. | Command Group | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `generate` | Submit generation jobs: `text-to-image`, `image-to-image`, `text-to-video`, `image-to-video`, `text-to-3d`, `image-to-3d`, `text-to-audio`, `voice`, `music`, `upscale`, `inpaint`, plus an `interactive` mode | | `jobs` | List, view status, cancel, monitor progress, and download outputs | | `workflows` | Create, list, version, execute, and manage workflows and templates | | `outputs` | Browse, search, filter, and download generated outputs | | `config` | Initialize CLI, set API profile, manage API keys and credentials | | `health` | Check service health for all Isis API endpoints | | `auth` | Login, logout, refresh tokens, and manage active sessions | **CLI Features:** - **Progress Bars** — Visual progress bars during generation jobs with percentage and estimated time remaining - **JSON Output Mode** — `--json` flag produces machine-readable JSON for use in shell scripts and CI pipelines - **API Key Authentication** — Configured via environment variable or `~/.isis/config.yaml`; also supports JWT bearer tokens - **Debug Mode** — `--debug` flag emits verbose request/response logging for troubleshooting - **Configurable Service URLs** — Override service URLs via `ISIS_GENERATION_API_URL`, `ISIS_WORKFLOW_REGISTRY_URL`, and `ISIS_OUTPUT_REGISTRY_URL`, or via the config file, for self-hosted deployments --- ## Client SDK ### `@isis/client` — TypeScript SDK The `IsisClient` provides a fully-typed programmatic interface for all Isis capabilities, suitable for use in other Oshun domain services or in external integrations. It is constructed with per-service URLs (the three REST services each have their own base URL) plus credentials. The `createClientFromEnv()` factory builds an instance from standard environment variables: `ISIS_GENERATION_API_URL`, `ISIS_WORKFLOW_REGISTRY_URL`, `ISIS_OUTPUT_REGISTRY_URL`, `ISIS_API_KEY`, and `ISIS_BEARER_TOKEN`. The following example shows the simplest usage path — submit a job and wait for its completion: ```typescript const client = createClientFromEnv(); // Submit a job and wait for completion const output = await client.submitAndWait({ type: 'text-to-image', prompt: 'A photorealistic ancient forest at golden hour', model: 'sdxl-1.0', parameters: { width: 1024, height: 1024, steps: 30 }, }); ``` **SDK Resources:** The client exposes four lazily-instantiated resource objects plus top-level delegate methods (`submitJob`, `getJob`, `waitForJob`, `submitAndWait`, …): - `client.generation` (`GenerationResource`) — `submitJob()`, `submitBatch()`, `getJob()`, `getJobStatus()`, `listJobs()`, `cancelJob()`, `retryJob()`, `getJobOutput()`, `getQueueStats()`, `getQueueHealth()`, dead-letter and backfill operations, webhook-subscription management - `client.workflows` (`WorkflowsResource`) — workflow `get()` plus the typed asset-pack recipe runners (`runCharacterConceptArtBaseline()`, etc.) - `client.outputs` (`OutputsResource`) — output listing, retrieval, download, lineage - `client.models` (`ModelsResource`) — model listing, retrieval, registration, upload **SDK Error Hierarchy:** `IsisError` is the base class. Subclasses are `NetworkError`, `TimeoutError`, `AbortError`, `RateLimitError`, `HttpError`, `ValidationError`, `ConfigError`, `JobError`, `WorkflowError`, and `OutputError`. Callers can catch specific error types to distinguish retryable from non-retryable failures. **SDK Features:** - Typed request and response objects for the generation modes - Automatic retry with backoff on transient failures - `RequestBuilder` for composing complex parameterized requests - Configurable request timeout --- ## Post-Production Studios (Phase 70) Six studio libraries close the remaining AI post-production gaps (`libs/isis/*`): - **`@isis/video-object-removal`** — mark-and-remove object erasure from video with temporally consistent inpainting (Phase 70.15). - **`@isis/relight-studio`** — generative relighting of footage and stills: light-direction/temperature edits after the fact (Phase 70.16). - **`@isis/video-edit-studio`** — instruction-driven video editing operations over existing footage (Phase 70.17). - **`@isis/portrait-studio`** — portrait animation / virtual performance: driving a still or reference face with performance video or audio (Phase 70.18). - **`@isis/foley-studio`** — foley and sound-design generation aligned to picture, beyond generic text-to-audio (Phase 70.19). - **`@isis/av-narrative-studio`** — joint audio-video multi-shot narrative generation with cross-shot consistency (Phase 70.20). --- ## Advanced 3D Generation Techniques (Phase 71.20–71.32) Thirteen technique libraries extend the core 3D generation surface (`libs/isis/*`): - **`@isis/3dgs-diffusion-editing`** — diffusion-based editing of 3D Gaussian splat scenes. - **`@isis/3dgs-relighting-pbr`** — relighting and PBR-material recovery for Gaussian splats. - **`@isis/3dgs-advanced`** — global illumination and reflection handling in splat rendering. - **`@isis/mesh-transformers`** — autoregressive mesh generation (MeshGPT/PolyGen-class token-based mesh synthesis). - **`@isis/score-distillation`** — SDS/VSD score-distillation optimization for text-to-3D. - **`@isis/fast-generation`** — flow-matching and consistency-model fast samplers for 3D generation. - **`@isis/3d-control`** — ControlNet/IP-Adapter-style conditioning for 3D generation (the 2D conditioning stack applied to 3D pipelines). - **`@isis/text-mesh-editing`** — natural-language mesh editing on existing assets. - **`@isis/3d-video-diffusion`** — video-diffusion priors for multi-view-consistent 3D. - **`@isis/hash-encoding`** — multiresolution hash-grid neural field encodings (Instant-NGP-class). - **`@isis/quad-mesh-gen`** — quad-dominant mesh generation and retopology. - **`@isis/part-level-3d`** — part-aware/segmented 3D generation for editable, articulated assets. - **`@isis/gigascale-3d`** — large-scene 3D generation beyond single-asset scale. GPU capacity for the self-hosted paths of these libraries dispatches through the standard RunPod serverless surface (`@isis/runpod-surface`, `@oshun/gpu-dispatcher`) — Phase 71.10. Where generation runs on the sovereign engine's GPU compute framework rather than RunPod, Isis is a co-owner of Phase 169 (GPU compute shader framework): it consumes the `@neith/compute-*` runtime (dispatch, memory, ML surface) for cross-backend GPU compute, with Neith owning the compute abstraction and Isis owning the generation workload. --- ## Cross-Domain Integration Isis is a pure capability domain. Other domains call its REST APIs directly to submit generation jobs; Isis broadcasts lifecycle events on the shared event bus so interested domains can react. Inbound jobs carry an opaque `projectId` that Isis stores verbatim and never resolves itself — the boundary exists specifically so Isis does not need to know anything about Yemaya projects, Hathor worlds, or Bellona assets. ### Events Published All Isis events are defined canonically as `IsisEventTypes` in `@oshun/contracts` and published through `@isis/event-publisher` onto `@oshun/event-bus` (Redis Streams). The full set: | Event | Trigger | | -------------------------- | --------------------------------------------- | | `isis.job.queued` | A new job is submitted and queued | | `isis.job.started` | A worker picks up a job and begins processing | | `isis.job.progress` | Periodic progress update during processing | | `isis.job.completed` | A job finishes successfully with outputs | | `isis.job.failed` | A job fails with an error | | `isis.job.cancelled` | A queued/running job is cancelled | | `isis.asset.generated` | A new asset is registered after generation | | `isis.workflow.registered` | A new workflow is created | | `isis.workflow.updated` | A workflow version is updated | | `isis.model.loaded` | A model is loaded onto a GPU worker | Each event has a typed Zod payload schema in `@oshun/contracts`. Event publishing is best-effort: a publish failure is logged and swallowed so it never breaks the generation flow. ### Inbound Requests Other domains submit work to Isis through its REST API (`POST /api/v1/jobs`, `POST /api/v1/workflows/:workflowId/run`) or the `@isis/client` SDK rather than through an event subscription. A job created on behalf of another domain carries that domain's `projectId`; the corresponding `isis.job.*` and `isis.asset.generated` events echo the `projectId` so the originating domain can correlate results back to its own records.