Historical task source. The Eve task board holds current task state.
Phase 78: Maya Forge AI Modding Companion — Voice-Interactive AI-Powered Modding Experience#
Purpose: Build the world's first voice-interactive, AI-powered modding companion with an expressive animated avatar and fully transparent workspace. Players and creators mod every aspect of any game at any granularity by conversing naturally with an embodied AI assistant that shows its work in a beautiful, interactive spatial display. This phase fuses Iris (AI agents), Psyche (avatar + voice), Maya Forge (modding framework), Isis (generative factory), and Hathor (narrative) into a single unified experience that has no equivalent in the industry.
Reference Analysis: Phase 78 report — AI modding companion vision & industry gap analysis (conversation record, 2026-03-19)
Scope: Voice conversation engine (streaming ASR/TTS, intent routing, spatial voice commands), expressive animated avatar (FACS expressions, lip sync, body language, emotion state machine, personality system), transparent workspace ("Glass Workshop" — spatial UI, live diffs, system graphs, reasoning stream, timeline, preview, impact visualization), universal mod engine abstraction (12 moddable domains across 4 granularity layers), real-time generation pipeline (text-to-3D/texture/animation/audio/code/narrative/UI/world with mid-generation feedback), safety and sandboxing layer, social and distribution integration (Maya Bazaar, Variants, Agora), learning and adaptation system, full integration orchestration, and end-to-end testing.
Technology:
- TypeScript libraries (in
libs/maya/forge-companion/): Orchestration layer, intent routing, mod abstraction, workspace UI, social integration, learning system - TypeScript libraries (in
libs/psyche/): Avatar controller, voice pipeline, expression engine, lip sync driver - TypeScript libraries (in
libs/iris/): Agent orchestration, memory, multi-turn dialogue, tool calling - Rust crates (in
libs/maya/engine-core/crates/): Spatial UI renderer, avatar animation runtime, voice audio pipeline, mod sandbox (WASM) - Integration targets: Psyche avatar system, Iris agent framework, Isis generative factory, Hathor narrative engine, Maya Forge mod framework, Maya Genesis procedural generation, Themis safety/IP, Euterpe audio, Bellona engine adapters
78.1 Voice Conversation Engine — Streaming ASR#
Real-time speech recognition optimized for noisy game environments with spatial audio awareness.
78.1.1 Core ASR Pipeline#
- 78.1.1.1 Create
libs/maya/forge-companion/voice-asr/library with project.json, package.json, tsconfig.json, and vitest config - 78.1.1.2 Implement
StreamingASREngineclass with configurable provider backends (OpenAI Whisper, AssemblyAI Universal-Streaming, Google Cloud Speech-to-Text V2, Deepgram Nova-3) - 78.1.1.3 Create
AudioCaptureManagerwith WebRTC getUserMedia integration, sample rate negotiation (16kHz/48kHz), and channel selection (mono voice isolation from stereo game audio) - 78.1.1.4 Implement
GameAudioNoiseFilterusing Web Audio API worklet — isolate voice from game SFX, music, and ambient using spectral gating, adaptive noise floor estimation, and voice activity detection - 78.1.1.5 Create
StreamingTranscriptAccumulatorthat assembles partial transcripts into coherent utterances with word-level timestamps and confidence scores - 78.1.1.6 Implement
VoiceActivityDetector(VAD) with configurable thresholds — detect speech onset within 50ms, speech end within 200ms, with hysteresis to prevent false triggers from game audio spikes - 78.1.1.7 Create
UtteranceBoundaryDetectorthat segments continuous speech into semantic utterances using pause detection (300ms default), prosodic cues (falling intonation), and syntactic completeness estimation - 78.1.1.8 Implement
ASRLatencyOptimizerwith streaming partial results — emit first partial within 150ms of speech onset, update every 100ms, finalize within 200ms of speech end - 78.1.1.9 Create
MultiLanguageASRsupporting 40+ languages with automatic language detection and mid-conversation language switching - 78.1.1.10 Implement
ASRConfidenceScorerthat rates transcript confidence per-word and per-utterance, flagging low-confidence segments for clarification - 78.1.1.11 Create
ASRProviderFailoverwith automatic fallback chain (primary → secondary → tertiary) with <100ms switchover on provider timeout or error spike - 78.1.1.12 Implement
ASRMetricsCollectortracking word error rate, latency percentiles (p50/p95/p99), provider availability, and noise rejection effectiveness - 78.1.1.13 Create comprehensive unit tests for AudioCaptureManager with mock MediaStream, noise filter with synthetic game audio + voice mixtures, VAD with edge cases (whispering, shouting, background music)
- 78.1.1.14 Create integration tests verifying end-to-end ASR pipeline latency <150ms with real audio samples in game-noise conditions
- 78.1.1.15 Implement ASR accuracy/latency benchmarks across game-audio
environments. (Blocker made visible 2026-09-18, from the removed HTML
comment: the provider seam + WER/latency collector are implemented in
78.1.1.x.) Needs a labelled real game-audio corpus (FPS combat, open-world
ambient, racing, etc.) with ground-truth transcripts to measure
comparative accuracy, and no such licensed corpus exists; the recorded
OpenRouter-ASR decision also supersedes benchmarking the paid
Whisper/AssemblyAI/Google/Deepgram vendors here (stated 2026-09-18).
blocked:corpus
78.1.2 Spatial Voice Commands#
- 78.1.2.1 Create
SpatialVoiceResolverthat maps deictic references ("this", "that", "here", "there", "over there") to game world coordinates using player camera raycast, cursor position, and gaze tracking - 78.1.2.2 Implement
GazeTrackerintegration with eye tracking APIs (Tobii, Meta Quest Pro eye tracking, webcam-based gaze estimation via MediaPipe Face Mesh) for determining player's visual focus point - 78.1.2.3 Create
PointingGestureDetectorusing hand tracking (Leap Motion, Meta Quest hand tracking, MediaPipe Hands) to resolve "that thing I'm pointing at" references - 78.1.2.4 Implement
EntityResolverthat identifies the game entity (object, NPC, terrain feature, UI element, building, item) at the resolved spatial coordinates via physics raycast and spatial query against the ECS - 78.1.2.5 Create
SpatialPronounStackmaintaining a ranked list of recently referenced entities so "it", "them", and "those" resolve correctly across multi-turn conversation - 78.1.2.6 Implement
RegionResolverfor area-based references — "this room", "this biome", "the forest over there" — using spatial partitioning (BSP, octree) and named region lookup - 78.1.2.7 Create
RelativeDirectionResolverfor "to the left", "behind me", "above the bridge" — computing direction relative to player's current orientation and position in world space - 78.1.2.8 Implement
GroupResolverfor plural references — "all the trees", "the guards", "everything in this area" — using entity type queries within spatial bounds - 78.1.2.9 Create
AmbiguityDisambiguatorthat detects when a spatial reference is ambiguous (multiple candidates) and generates a clarification question with visual highlighting of candidates - 78.1.2.10 Implement unit tests for spatial resolution across 20+ deictic patterns with mock game world geometry and entity placement
78.1.3 Intent Classification & Entity Extraction#
- 78.1.3.1 Create
ModIntentClassifierwith 12 primary intent categories matching Maya Forge's mod taxonomy: asset, code, rule, world, ui, narrative, physics, ai-behavior, audio, economy, social, accessibility - 78.1.3.2 Implement
IntentHierarchywith 48 sub-intents — each primary category has 4 sub-intents (create, modify, remove, query) mapped to specific mod operations - 78.1.3.3 Create
EntityExtractorthat identifies mod-relevant entities from transcribed speech: game objects, property names, numeric values, colors, directions, comparisons ("bigger", "faster", "more"), temporal references ("at sunset", "when the player enters") - 78.1.3.4 Implement
GranularityDetectorthat classifies the user's expertise level from their language — novice ("make it cooler"), intermediate ("adjust the particle emission rate"), expert ("modify the Rayleigh scattering coefficient") — and routes to appropriate abstraction layer - 78.1.3.5 Create
CompoundCommandParserfor multi-part requests — "make the sky purple AND add rain AND make the NPCs run inside" — splitting into ordered sub-commands with dependency detection - 78.1.3.6 Implement
ConditionalCommandParserfor conditional mods — "when it rains, make the roads slippery" — extracting trigger conditions and mod actions as event-driven rules - 78.1.3.7 Create
UndoRedoIntentDetectorrecognizing undo/redo requests: "go back", "undo that", "no not like that", "revert", "actually keep the first version", "try again but different" - 78.1.3.8 Implement
ClarificationGeneratorthat produces natural language clarification questions when intent is ambiguous, with suggested options: "Did you mean the wolf's hunting behavior or its appearance?" - 78.1.3.9 Create
IntentConfidenceThresholdsystem — high confidence (>0.9) executes immediately, medium (0.7-0.9) confirms before executing, low (<0.7) asks for rephrasing - 78.1.3.10 Implement intent classification unit tests with 500+ labeled utterances across all 12 categories and 4 granularity levels
- 78.1.3.11 Create intent classification accuracy benchmark targeting >95% top-1 accuracy on the labeled test set
- 78.1.3.12 Implement entity extraction tests verifying correct parsing of numeric values, colors (named and hex), spatial references, temporal references, and comparative modifiers
78.2 Voice Conversation Engine — TTS & Dialogue Management#
Expressive text-to-speech with personality, emotion modulation, and multi-turn dialogue state management.
78.2.1 Streaming TTS Pipeline#
- 78.2.1.1 Create
libs/maya/forge-companion/voice-tts/library with configurable TTS provider backends (ElevenLabs, OpenAI TTS, Google Cloud TTS, CosyVoice2, Chatterbox) - 78.2.1.2 Implement
StreamingTTSEnginewith chunked text input — begin synthesis from first sentence while LLM is still generating remaining response, achieving <200ms first-audio-byte latency - 78.2.1.3 Create
EmotionTagParserthat reads emotion annotations from LLM output (joy, concern, excitement, curiosity, playfulness, caution, pride, empathy) and maps to TTS SSML prosody parameters (pitch, rate, volume, emphasis) - 78.2.1.4 Implement
PersonalityVoiceProfilesystem — each avatar personality has a distinct voice configuration (pitch range, speaking rate, vocal timbre, laugh style, filler words, catchphrases) - 78.2.1.5 Create
AdaptiveSpeechRatethat adjusts speaking speed based on content complexity — slows down for technical explanations, speeds up for confirmations and acknowledgments - 78.2.1.6 Implement
PhonemeStreamEmitterthat extracts phoneme timing data from TTS output in real-time for driving avatar lip sync — emit phoneme events with 10ms precision timestamps - 78.2.1.7 Create
ProsodyAnalyzerthat extracts pitch contour, energy envelope, and speech rhythm from generated audio for driving avatar expression and gesture timing - 78.2.1.8 Implement
AudioOutputRoutermanaging spatial audio positioning — avatar voice emanates from avatar's world position using HRTF, with configurable attenuation and occlusion - 78.2.1.9 Create
TTSCacheManagercaching frequently used phrases ("Got it", "On it", "Let me think about that") for instant playback without synthesis latency - 78.2.1.10 Implement
VoiceConsistencyGuardensuring voice identity remains stable across sessions using voice embedding comparison (Psyche voice-consistency integration) - 78.2.1.11 Create
TTSProviderFailoverwith seamless provider switching maintaining voice character consistency across providers - 78.2.1.12 Implement TTS unit tests verifying emotion tag parsing, phoneme stream timing accuracy, and adaptive rate adjustments
- 78.2.1.13 Create TTS latency benchmarks — first audio byte <200ms, phoneme stream alignment <15ms drift from audio playback
78.2.2 Multi-Turn Dialogue Management#
- 78.2.2.1 Create
ModDialogueManagerintegrating Iris agent framework with mod-specific dialogue state — tracks current mod operation, pending confirmations, undo stack, and conversation topic - 78.2.2.2 Implement
ConversationContextWindowmaintaining last 20 turns of dialogue with entity references, spatial context, and mod operations for LLM context injection - 78.2.2.3 Create
InterruptionHandlersupporting barge-in — when player speaks while avatar is talking, gracefully interrupt TTS, process new input, and resume or redirect based on content - 78.2.2.4 Implement
InterruptionClassifierdistinguishing between corrections ("no, I meant blue"), additions ("and also make it glow"), and topic changes ("actually forget that, let's work on the UI") - 78.2.2.5 Create
ProactiveDialogueEnginewhere the avatar proactively offers suggestions based on context — "I notice you've been adjusting gravity a lot — want me to create a gravity zone system instead?" - 78.2.2.6 Implement
ConfirmationProtocolwith risk-proportional confirmation — trivial mods (color change) apply instantly with verbal confirmation, significant mods (physics overhaul) require explicit "yes" before applying - 78.2.2.7 Create
DialogueMemoryBridgeconnecting conversation state to Iris long-term memory — important preferences, decisions, and patterns persist across sessions - 78.2.2.8 Implement
MultiModalInputFusioncombining voice, gaze, gesture, and direct manipulation (clicking/dragging in workspace) into a unified intent stream — "make THAT [points] like THIS [adjusts slider]" - 78.2.2.9 Create
NaturalAcknowledgmentGeneratorproducing varied, personality-consistent acknowledgments instead of robotic "Understood" — e.g., "Ooh nice idea", "Coming right up", "Let's see how that looks" - 78.2.2.10 Implement dialogue management unit tests covering 30+ multi-turn scenarios including interruptions, corrections, topic changes, compound commands, and undo sequences
78.3 Expressive Animated Avatar — Facial Expression Engine#
FACS-based facial animation with procedural micro-expressions, emotion-driven blending, and real-time responsiveness.
78.3.1 FACS Blend Shape System#
- 78.3.1.1 Create
libs/maya/forge-companion/avatar-controller/library as the central avatar orchestration layer binding Psyche avatar subsystems to the forge companion context - 78.3.1.2 Implement
FACSBlendShapeDrivermapping 46 FACS Action Units to 250+ blend shapes (Psyche avatar-expressions integration) with per-AU intensity curves and asymmetric left/right control - 78.3.1.3 Create
EmotionToFACSMappertranslating continuous emotion state (valence-arousal-dominance model) to FACS AU activation patterns — joy → AU6+AU12, surprise → AU1+AU2+AU5+AU26, concern → AU1+AU4+AU15 - 78.3.1.4 Implement
MicroExpressionGeneratorproducing subtle, involuntary facial movements (50-500ms duration) that leak emotional state — slight brow furrow when processing a complex request, lip corner twitch when amused - 78.3.1.5 Create
ExpressionTransitionEnginewith smooth easing between emotional states — never snap from happy to concerned, always transition through intermediate states with configurable duration (200-800ms default) - 78.3.1.6 Implement
ExpressionLayerCompositorsupporting simultaneous expression layers — base emotion + speech emphasis + reactive surprise + micro-expressions, blended by priority and intensity - 78.3.1.7 Create
BlinkControllerwith naturalistic blink patterns — average 15-20 blinks/minute, rate increases during speech and stress, decreases during focused listening, with occasional double-blinks - 78.3.1.8 Implement
EyeGazeControllerwith saccadic movement patterns — avatar looks at player during conversation, glances at workspace elements when referencing them, breaks gaze naturally during thinking - 78.3.1.9 Create
PupilDilationSimulatorresponding to emotional arousal (excitement → dilate, focus → constrict) and ambient lighting conditions in the game world - 78.3.1.10 Implement
ExpressionPresetLibrarywith 60+ named expression presets (delighted, skeptical, impressed, puzzled, mischievous, determined, sympathetic, etc.) as starting points for procedural variation
78.3.2 Lip Sync System#
- 78.3.2.1 Create
RealtimeLipSyncDriverconsuming phoneme stream from TTS pipeline and driving viseme blend shapes with <15ms audio-visual sync accuracy (Psyche avatar-lipsync integration) - 78.3.2.2 Implement
VisemeMappermapping 44 English phonemes to 15 core visemes plus 8 co-articulation variants, with language-specific viseme sets for 12 additional languages - 78.3.2.3 Create
CoArticulationBlenderthat smooths transitions between visemes based on preceding and following phonemes — "pa" differs from "ta" in anticipatory lip rounding - 78.3.2.4 Implement
EmotionalLipSyncModifieradjusting lip shape based on emotional state — smiling while speaking lifts lip corners, concerned speech narrows lip opening - 78.3.2.5 Create
SpeechEmphasisAnimatordetecting stressed syllables and amplifying jaw opening, lip spread, and head movement on emphasized words - 78.3.2.6 Implement
SilentMouthMovementfor non-speech mouth behaviors — idle lip licking, slight jaw clench when thinking, mouth opening slightly before speaking (anticipatory gesture) - 78.3.2.7 Create
LipSyncQualityMonitortracking audio-visual sync drift in real-time and applying correction offsets to maintain <15ms accuracy even under system load - 78.3.2.8 Implement lip sync unit tests with 100+ phoneme sequences verifying viseme selection, transition timing, and co-articulation accuracy
78.4 Expressive Animated Avatar — Body Language & Animation#
Procedural body animation driven by speech content, emotion, and conversational context.
78.4.1 Gesture Generation System#
- 78.4.1.1 Create
ProceduralGestureEnginegenerating hand and arm gestures synchronized to speech — beat gestures on stressed syllables, deictic gestures when referencing objects, iconic gestures representing shapes/sizes/directions - 78.4.1.2 Implement
GestureLibrarywith 120+ motion-captured gesture clips organized by category: conversational (shrug, nod, head tilt), illustrative (pointing, sweeping, sizing), emotional (fist pump, face palm, chin stroke), and workspace interaction (grabbing, dragging, swiping) - 78.4.1.3 Create
GestureTriggerSystemmapping semantic content to gesture categories — mentions of direction trigger pointing, mentions of size trigger scaling gestures, mentions of lists trigger enumerating finger gestures - 78.4.1.4 Implement
GestureTimingAlignersynchronizing gesture onset to speech prosody peaks — gesture stroke lands on the stressed syllable of the keyword it illustrates - 78.4.1.5 Create
GestureBlenderwith IK-based blending between gesture clips and idle pose — smooth entry/exit, natural arm paths, no interpenetration with avatar body - 78.4.1.6 Implement
WorkspaceInteractionGestureswhere the avatar physically interacts with Glass Workshop elements — "pulls" code panels into view, "swipes" through timeline, "pinches" to zoom system graph, "taps" on parameters to highlight them - 78.4.1.7 Create
HeadMotionControllerwith conversational head movement — nods during listening (agreement/acknowledgment), head tilts during curiosity, head shakes during disagreement/impossibility, chin raise during confidence - 78.4.1.8 Implement
PostureStateManagerwith full-body posture states — leaning forward (engaged), leaning back (relaxed/contemplative), arms crossed (thinking), open stance (explaining), turned toward workspace (working)
78.4.2 Idle & Ambient Animation#
- 78.4.2.1 Create
IdleAnimationSystemwith procedural idle behaviors — weight shifting (every 5-15s), subtle swaying (Perlin noise-driven), breathing (12-20 breaths/min with emotional rate modulation), occasional stretching - 78.4.2.2 Implement
BreathingControllerwith visible chest/shoulder rise-fall synchronized to speech pauses, deeper breaths before long explanations, quick breath on surprise - 78.4.2.3 Create
EnvironmentReactionSystem— avatar reacts to game world events: flinches at nearby explosions, shields eyes from bright flashes, looks toward loud sounds, shivers in cold environments, wipes brow in hot ones - 78.4.2.4 Implement
AttentionDistributionControllergoverning where the avatar looks when not directly engaged — glances at interesting game events, inspects nearby objects, watches the player's current activity - 78.4.2.5 Create
FidgetBehaviorGeneratorwith personality-appropriate idle fidgets — adjusts virtual glasses, plays with hair, taps fingers, examines own hands, hums quietly (with audio) - 78.4.2.6 Implement
ClothAndHairSimulationintegration — avatar clothing and hair respond to movement, wind, and animation with physically-based secondary motion
78.4.3 Emotional State Machine#
- 78.4.3.1 Create
ContinuousEmotionModelusing valence-arousal-dominance (VAD) 3D emotional space — no discrete emotion states, but continuous trajectories through emotional space with momentum and decay - 78.4.3.2 Implement
EmotionDrivers— input sources that shift emotional state: conversation sentiment (player praise → joy, player frustration → empathy), mod success (satisfaction), mod failure (concern), complexity (focused intensity), humor detection (amusement) - 78.4.3.3 Create
EmotionDecayFunctionwith natural emotional return to baseline — strong emotions decay over 10-30s, mild shifts decay over 3-5s, with personality-dependent decay rates (some personalities are more emotionally stable, others more volatile) - 78.4.3.4 Implement
EmotionToAnimationMappertranslating VAD coordinates to animation parameter sets: facial expression intensities, gesture frequency/amplitude, posture openness, movement energy, voice emotion tags - 78.4.3.5 Create
MoodTrackermaintaining a slow-moving mood state (hours-scale) that biases moment-to-moment emotions — after a productive session, baseline mood is happier; after many failures, baseline shifts to more cautious/supportive - 78.4.3.6 Implement emotion system unit tests verifying smooth transitions, decay behavior, driver responses, and personality variation across 5 personality archetypes
78.5 Expressive Animated Avatar — Personality & Customization#
Configurable avatar personality system with player-driven appearance and behavior customization.
78.5.1 Personality Framework#
- 78.5.1.1 Create
PersonalityEngineimplementing Big Five personality traits (openness, conscientiousness, extraversion, agreeableness, neuroticism) as continuous 0-1 parameters that influence all avatar behaviors - 78.5.1.2 Implement
PersonalityToVoiceMapper— extraversion affects speaking rate and volume, agreeableness affects warmth and softness, neuroticism affects pitch variability, openness affects vocabulary diversity, conscientiousness affects precision - 78.5.1.3 Create
PersonalityToGestureMapper— extraverts gesture more broadly, introverts gesture more subtly, agreeable personalities lean in more, conscientious personalities use more precise/measured gestures - 78.5.1.4 Implement
PersonalityToDialogueMapper— affects word choice ("Awesome!" vs "Satisfactory"), humor frequency, explanation depth, proactive suggestion rate, and confirmation protocol strictness - 78.5.1.5 Create
PersonalityPresetswith 8 curated personality archetypes: The Enthusiast (high E, high O), The Professor (high C, high O), The Guardian (high A, high C), The Trickster (high O, low C), The Artisan (high O, high A), The Commander (low A, high C), The Sage (low E, high O), The Companion (high A, high E) - 78.5.1.6 Implement
PersonalityEvolutionallowing personality to subtly shift based on player interaction patterns — a player who appreciates humor sees the avatar gradually become funnier, a player who values precision sees the avatar become more methodical - 78.5.1.7 Create personality unit tests verifying that each Big Five trait measurably affects voice, gesture, dialogue, and expression outputs
78.5.2 Appearance Customization#
- 78.5.2.1 Create
AvatarAppearanceEditorUI allowing players to customize their companion's visual appearance — body type, face shape, skin tone, hair style/color, eye color, clothing style - 78.5.2.2 Implement
AvatarStylePresetswith 12 curated visual styles: realistic human, stylized anime, fantasy creature, sci-fi android, cosmic entity, animal companion, geometric abstract, steampunk automaton, nature spirit, shadow figure, hologram, chibi - 78.5.2.3 Create
AvatarClothingSystemwith outfit categories that reflect personality — formal/professional, casual/relaxed, fantasy/armored, sci-fi/techy, artistic/bohemian — with dynamic clothing changes based on current task context - 78.5.2.4 Implement
AvatarAccessorySystem— glasses, hats, jewelry, tools (wrench for mechanics mods, paintbrush for visual mods, wand for magic mods) that change based on current modding domain - 78.5.2.5 Create
AvatarScaleController— avatar can be full-size standing beside player, miniature shoulder companion, floating head-only, or full-screen face during important conversations - 78.5.2.6 Implement
AvatarPersistencesaving appearance, personality, name, and learned preferences to player profile with cross-session continuity
78.6 Transparent Workspace — Glass Workshop Core#
Spatial UI framework for the "Glass Workshop" — a beautiful, interactive, transparent display showing everything the AI is doing.
78.6.1 Spatial UI Framework#
- 78.6.1.1 Create
libs/maya/forge-companion/glass-workshop/library as the transparent workspace rendering and interaction system - 78.6.1.2 Implement
SpatialPanelManagermanaging floating UI panels in 3D space around the avatar — panels orbit, dock, stack, and arrange based on content type and player focus - 78.6.1.3 Create
PanelRenderersupporting multiple panel types: code editor, diff viewer, node graph, preview viewport, timeline, property inspector, console output, progress indicator - 78.6.1.4 Implement
PanelLayoutEnginewith automatic spatial arrangement — panels don't overlap, maintain readable angle to player camera, respect comfortable viewing distance (1-3m in VR, screen-relative in flat) - 78.6.1.5 Create
PanelTransitionAnimatorwith smooth panel entrance/exit animations — panels materialize from particles, slide in from edges, expand from avatar's hands, or fold like origami - 78.6.1.6 Implement
PanelFocusSystem— when avatar references a panel, it highlights, enlarges slightly, and orients toward player; unfocused panels dim and recede - 78.6.1.7 Create
PanelInteractionHandlersupporting both voice commands ("close that panel", "zoom in on the code") and direct manipulation (click/drag to move, pinch to resize, swipe to scroll) - 78.6.1.8 Implement
WorkshopThemeEnginewith visual themes that match the game's aesthetic — fantasy game gets scroll/parchment panels, sci-fi gets holographic panels, modern gets glassmorphism, horror gets flickering/glitchy panels - 78.6.1.9 Create
WorkshopOpacityControllerallowing players to adjust overall workshop transparency from fully opaque (focused work) to nearly transparent (ambient monitoring) to hidden (avatar only) - 78.6.1.10 Implement
VRWorkshopAdapterfor XR environments — panels are true 3D objects in space, grabbable with controllers, arrangeable in 360 degrees around the player - 78.6.1.11 Create
FlatScreenWorkshopAdapterfor traditional displays — panels rendered as overlay windows with configurable screen regions (side panel, bottom bar, picture-in-picture) - 78.6.1.12 Implement spatial UI unit tests verifying panel layout correctness, transition smoothness, focus system accuracy, and interaction handler responsiveness
78.6.2 Live Diff Viewer#
- 78.6.2.1 Create
LiveDiffPanelshowing real-time diffs of every change the AI makes — config values, code modifications, asset substitutions, parameter adjustments — with syntax highlighting and inline annotations - 78.6.2.2 Implement
DiffStreamRendererthat renders diffs incrementally as changes are applied — each line appears as the AI writes it, with a cursor effect showing progress - 78.6.2.3 Create
DiffCategoryColorCodingwith color-coded diff categories: additions (green), removals (red), modifications (amber), moved (blue), generated (purple), with configurable colorblind-friendly palette - 78.6.2.4 Implement
DiffAnnotationLayerwhere the avatar's explanation text appears as inline annotations next to the relevant diff lines — "I changed this because..." - 78.6.2.5 Create
DiffApprovalControls— player can approve individual changes, reject specific lines, or approve/reject all with voice or click - 78.6.2.6 Implement
DiffHistoryNavigatorallowing scrolling through all diffs in the current session with jump-to-time functionality
78.6.3 System Dependency Graph#
- 78.6.3.1 Create
SystemGraphPanelrendering an interactive node graph of all game systems affected by the current mod — nodes represent systems (physics, rendering, AI, economy, etc.), edges represent dependencies - 78.6.3.2 Implement
AffectedNodeHighlighterthat pulses/glows nodes currently being modified, with ripple animations showing propagation of changes through dependent systems - 78.6.3.3 Create
NodeDetailPopover— hovering/focusing on a node shows its current state, what's being changed, and estimated impact - 78.6.3.4 Implement
GraphFilterControls— filter by system category, impact severity, or change type; collapse unaffected subtrees for clarity - 78.6.3.5 Create
DependencyWarningOverlayhighlighting dangerous dependency chains — "changing physics gravity will affect 3 quest triggers that rely on fall distance calculations"
78.6.4 Reasoning Stream#
- 78.6.4.1 Create
ReasoningStreamPaneldisplaying a curated, beautiful representation of the AI's decision-making process — not raw chain of thought, but an artful visualization of the reasoning steps - 78.6.4.2 Implement
ReasoningStepRendererwith step types: analysis (magnifying glass icon), decision (fork icon), action (lightning icon), verification (checkmark icon), rollback (rewind icon) - 78.6.4.3 Create
ReasoningAnimations— each step appears with a characteristic animation: analysis spirals in, decisions branch like a tree, actions strike like lightning, verifications stamp with a seal - 78.6.4.4 Implement
ReasoningVerbosityControlwith three levels: minimal (only decisions and actions), standard (full reasoning chain), detailed (includes alternatives considered and why they were rejected) - 78.6.4.5 Create
ReasoningToSpeechBridge— the avatar narrates key reasoning steps naturally: "I'm looking at the physics system... ah, I see the gravity is coupled to the jump height calculator, so I'll need to adjust both..."
78.7 Transparent Workspace — Timeline, Preview & Impact#
Advanced workspace panels for temporal navigation, live preview, and impact assessment.
78.7.1 Timeline & Rollback System#
- 78.7.1.1 Create
ModTimelinePanelvisualizing all mod operations in the current session as an interactive horizontal timeline with thumbnails/icons per operation - 78.7.1.2 Implement
TimelineScrubber— drag to any point in the timeline to preview the game state at that moment; voice command "go back to before I changed the gravity" navigates by semantic description - 78.7.1.3 Create
TimelineBranchManagersupporting branching timelines — "try this a different way" creates a branch; "compare the two approaches" shows side-by-side - 78.7.1.4 Implement
SnapshotEnginecapturing full game state snapshots at configurable intervals and before every mod operation — snapping ECS state, config values, asset references, script state - 78.7.1.5 Create
SelectiveRollbackallowing rolling back specific changes while keeping others — "undo the gravity change but keep the sky color" — using dependency analysis to detect conflicts - 78.7.1.6 Implement
TimelineAnnotation— player can add voice notes to timeline points: "this is the version I liked for the demo" - 78.7.1.7 Create
TimelineExportImportfor saving/loading timeline sessions as mod packages with full provenance - 78.7.1.8 Implement timeline unit tests covering branching, selective rollback with dependency conflicts, and snapshot integrity
78.7.2 Live Preview System#
- 78.7.2.1 Create
LivePreviewPanelshowing a real-time preview of the mod's effect before it's applied to the main game world — runs in an isolated preview sandbox - 78.7.2.2 Implement
SplitScreenPreviewwith configurable split (left/right, top/bottom, picture-in-picture) showing before/after comparison in real-time - 78.7.2.3 Create
PreviewCameraControllerwith independent camera in the preview pane — player can orbit, zoom, and navigate to see the mod from different angles without affecting main camera - 78.7.2.4 Implement
PreviewTimeControlwith play/pause/slow-mo/fast- forward in the preview to see time-dependent mods (day-night cycle changes, weather mods, AI behavior over time) - 78.7.2.5 Create
PreviewStressTest— automatically simulate edge cases in preview: 100 NPCs in the scene, extreme weather, rapid day-night cycling, to surface potential issues before applying - 78.7.2.6 Implement
PreviewToApplyTransitionwith a satisfying visual transition when applying the previewed mod — preview expands to fill the main viewport, or the change ripples outward from the preview panel
78.7.3 Impact Assessment Visualizer#
- 78.7.3.1 Create
ImpactAssessmentEngineanalyzing a proposed mod's effects across all game systems — physics, economy, narrative, AI, performance, accessibility — and producing a structured impact report - 78.7.3.2 Implement
ImpactVisualizationPanelwith concentric rings showing direct effects (inner), secondary effects (middle), and tertiary ripple effects (outer) with severity color coding - 78.7.3.3 Create
PerformanceBudgetMetershowing estimated FPS impact, memory delta, draw call change, and physics tick cost of the proposed mod with green/yellow/red thresholds - 78.7.3.4 Implement
BalanceImpactEstimatoranalyzing how the mod affects game balance — difficulty curves, economy equilibrium, combat fairness — using simulation rollouts - 78.7.3.5 Create
NarrativeCoherenceCheckerverifying the mod doesn't break quest logic, dialogue continuity, or world lore — flags conflicts like "this NPC can't be removed because 3 quests reference them" - 78.7.3.6 Implement
AccessibilityImpactScannerchecking whether the mod degrades accessibility — contrast ratios, font sizes, audio cue alternatives, colorblind safety - 78.7.3.7 Create
MultiplayerCompatibilityCheckerassessing whether the mod is compatible with multiplayer — server-authoritative validation, bandwidth impact, desync risk - 78.7.3.8 Implement impact assessment unit tests with 20+ mod scenarios verifying correct cross-system impact detection and severity classification
78.7.4 Confidence & Progress Display#
- 78.7.4.1 Create
ConfidenceMeter— a visual indicator showing the AI's confidence in its current approach (0-100%), rendered as an animated gauge, color gradient, or avatar expression intensity - 78.7.4.2 Implement
ConfidenceToActionMapping— high confidence (>90%) means the avatar works smoothly and quickly; medium (70-90%) means it pauses to explain its reasoning; low (<70%) means it stops and asks for guidance with specific questions - 78.7.4.3 Create
ProgressTrackershowing completion percentage of the current mod operation with sub-task breakdown — "Adjusting physics (done) → Updating quest triggers (in progress) → Testing edge cases (pending)" - 78.7.4.4 Implement
ProgressToAvatarBehaviorMapper— in-progress work shows the avatar actively working (hands moving, focused expression); blocked work shows the avatar pausing and looking at player; completed work shows satisfaction/pride
78.8 Universal Mod Engine — Abstraction Layer#
The four-layer abstraction stack translating natural language into engine primitives, supporting all 12 moddable domains.
78.8.1 Abstraction Stack Core#
- 78.8.1.1 Create
libs/maya/forge-companion/mod-abstraction/library implementing the four-layer mod abstraction stack - 78.8.1.2 Implement
Layer4NaturalLanguageinterface — accepts freeform text/speech input and produces structuredModIntentwith extracted entities, spatial references, and granularity level - 78.8.1.3 Create
Layer3SemanticActions— a domain-specific action vocabulary (SetAtmosphere, SpawnEntity, AdjustPhysics, ModifyBehavior, AlterEconomy, EditDialogue, etc.) with typed parameters and validation - 78.8.1.4 Implement
Layer2SystemCommands— engine-level commands that map to specific subsystem APIs (physics.setGravity, renderer.setShader, ecs.addComponent, navmesh.rebuild) - 78.8.1.5 Create
Layer1EnginePrimitives— raw engine operations (shader uniform sets, ECS component mutations, buffer uploads, config writes) that execute atomically - 78.8.1.6 Implement
LayerTranslator— the LLM-powered component that translates between adjacent layers, with validation at each step and rollback on failure - 78.8.1.7 Create
GranularityRouter— routes novice speech to Layer 4 translation, intermediate to Layer 3, expert to Layer 2/1 direct access, with automatic detection of the user's intended granularity - 78.8.1.8 Implement
ModOperationTransactionwrapping every mod in a transaction with commit/rollback semantics — partial failures roll back completely, successful mods commit atomically - 78.8.1.9 Create
ModValidationPipelinevalidating each mod at every layer — semantic validity (Layer 3), system consistency (Layer 2), engine safety (Layer 1) — before committing - 78.8.1.10 Implement abstraction layer unit tests with 100+ mod operations verifying correct translation through all 4 layers across all 12 domains
78.8.2 Asset Modding Domain#
- 78.8.2.1 Create
AssetModHandlersupporting: texture replacement, model swapping, material property adjustment, shader assignment, LOD configuration, particle effect modification, decal placement - 78.8.2.2 Implement
AssetReferenceResolvermapping natural language asset descriptions ("the sword the boss drops", "all the trees in the forest") to specific asset IDs via game metadata and spatial queries - 78.8.2.3 Create
AssetPropertyEditorexposing every material property (albedo, roughness, metallic, normal intensity, emission, opacity, refraction), every mesh property (scale, pivot, collision shape), and every particle property (emission rate, lifetime, velocity, color curve) - 78.8.2.4 Implement
AssetPreviewInContext— render the modified asset inside a representative scene (not an isolated viewer) through the Maya engine's headless renderer (the phase-79 79.0.1 wgpu harness: lavapipe on the Linux server, Metal on the Mac; install-firstmesa-vulkan-drivers, park only with a named failed attempt). (Blocker made visible 2026-09-18: the prior HTML comment said this needs a running renderer — the Maya engine is that renderer.) Verify: a spec swaps a fixture asset into a fixture scene, renders a frame headless and asserts the swapped asset appears at its placement (compare against a golden and against the pre-swap frame). - 78.8.2.5 Create asset-modding integration tests verifying texture hotswap, material-property animation and multi-asset batch operations, driving the asset handler + reference resolver + property editor (78.8.2.1-.3) through the headless render harness of 78.8.2.4. (Blocker made visible 2026-09-18: the prior HTML comment covering .4/.5 said these need a running renderer.) Verify: a spec hotswaps a texture and asserts the rendered pixels change, animates a material property over frames and asserts the change, and runs an "all the trees" batch and asserts every matched instance updated.
78.8.3 Code & Script Modding Domain#
- 78.8.3.1 Create
ScriptModHandlersupporting: new script creation, existing script modification, event handler registration, component behavior extension, API hook injection - 78.8.3.2 Implement
ScriptGeneratorusing LLM code generation to produce Luau, TypeScript, or Lua scripts from natural language descriptions with correct API usage, type safety, and error handling - 78.8.3.3 Create
ScriptSandboxTesterthat automatically tests generated scripts in an isolated sandbox — runs unit tests, checks for infinite loops, validates memory usage, and catches runtime errors before deployment - 78.8.3.4 Implement
ScriptExplainerthat can explain any existing game script in natural language at the player's expertise level — novice gets "this makes enemies chase you", expert gets "this implements a utility-based AI with distance-weighted threat evaluation" - 78.8.3.5 Create
ScriptDiffReviewershowing proposed script changes with inline explanations and allowing line-by-line approval
78.8.4 Rules & Mechanics Modding Domain#
- 78.8.4.1 Create
RulesModHandlersupporting: numeric parameter adjustment (damage values, speeds, cooldowns), formula modification (damage = base _ multiplier + bonus → damage = base ^ 1.2 _ level), new rule creation, condition/trigger definition - 78.8.4.2 Implement
GameBalanceSimulatorthat runs Monte Carlo simulations of modified rules — test 1000 combat encounters with new damage formula, simulate 100 economic cycles with new prices — and reports statistical outcomes - 78.8.4.3 Create
RuleChainVisualizershowing how rules interact — "this damage formula feeds into this resistance calculation, which feeds into this health system, which triggers this death handler" - 78.8.4.4 Implement
ParameterTweakerwith real-time slider controls generated from any numeric parameter — player adjusts slider, sees effect live in preview, avatar explains trade-offs
78.8.5 World & Environment Modding Domain#
- 78.8.5.1 Create
WorldModHandlersupporting: terrain sculpting, biome placement, weather system adjustment, day-night cycle modification, atmosphere tuning, water body creation, vegetation distribution - 78.8.5.2 Implement
TerrainVoiceSculptortranslating voice commands to terrain operations — "raise a mountain here", "carve a river from here to there", "make this area a desert" — using Genesis terrain generation (Maya genesis-terrain integration) - 78.8.5.3 Create
BiomeDesignerallowing voice-driven biome definition — "this area should be a dark forest with giant mushrooms and bioluminescent plants" — generating terrain, vegetation, lighting, ambient audio, and weather as a coherent package - 78.8.5.4 Implement
AtmosphereComposerfor voice-driven atmospheric effects — "make the sunset more dramatic", "add fog in the valley", "the sky should have aurora borealis at night" - 78.8.5.5 Create world-modding integration tests verifying terrain
generation, biome coherence and atmosphere rendering, driving
BiomeDesigner + TerrainVoiceSculptor + AtmosphereComposer (78.8.5.1-.4)
through the Maya Genesis pipeline (
libs/maya/genesis-terrain,genesis-flora) and the headless render harness. (Blocker made visible 2026-09-18: the prior HTML comment said terrain gen needs the Genesis pipeline and atmosphere needs a renderer — both are in-repo.) Verify: a spec generates a fixture terrain+biome, asserts biome adjacency rules hold and vegetation density matches the biome, and renders the atmosphere headless asserting the sky parameters took effect.
78.8.6 UI/UX Modding Domain#
- 78.8.6.1 Create
UIModHandlersupporting: HUD element repositioning/ resizing/restyling, new UI widget creation, minimap customization, menu redesign, tooltip modification, notification system changes - 78.8.6.2 Implement
UIComponentGeneratorproducing custom UI components from voice descriptions — "add a compass at the top of the screen that shows quest markers" — generating component code, styling, and data bindings - 78.8.6.3 Create
UILayoutEditorwith voice-driven layout commands — "move the health bar to the bottom left", "make the inventory grid 6 columns instead of 4", "add a damage numbers popup" - 78.8.6.4 Implement
UIStyleTransfer— "make the UI look like [reference game]" — applying style characteristics (color palette, font family, border style, animation timing) from reference descriptions
78.8.7 Narrative & Dialogue Modding Domain#
- 78.8.7.1 Create
NarrativeModHandlersupporting: dialogue editing, new dialogue branch creation, quest modification, lore addition, NPC personality adjustment, story arc modification (Hathor integration) - 78.8.7.2 Implement
DialogueTreeEditorwith voice-driven dialogue authoring — "add an option where the player can ask about the dragon" — generating dialogue nodes, response options, condition gates, and consequence triggers - 78.8.7.3 Create
NPCPersonalityEditorallowing voice-driven NPC personality changes — "make this shopkeeper grumpy but secretly kind" — adjusting dialogue tone, greeting variations, response tendencies, and hidden loyalty thresholds - 78.8.7.4 Implement
QuestGraphEditorwith voice-driven quest modification — "add a secret path to complete this quest by befriending the boss instead of fighting" — generating alternative objective chains, new dialogue, and reward branches - 78.8.7.5 Create
LoreConsistencyEnginethat maintains narrative coherence across all player-created narrative mods, flagging contradictions with existing lore and suggesting resolutions
78.8.8 Physics Modding Domain#
- 78.8.8.1 Create
PhysicsModHandlersupporting: gravity adjustment (global and per-zone), friction coefficients, material density, collision behavior, joint constraints, fluid viscosity, ragdoll tuning, buoyancy - 78.8.8.2 Implement
PhysicsZoneCreator— voice-driven creation of spatial physics zones: "zero gravity inside this building", "double gravity in the training arena", "moon gravity on the mountain top" — with smooth boundary transitions - 78.8.8.3 Create
PhysicsMaterialDesigner— "make ice actually slippery", "this surface should be bouncy like a trampoline", "metal objects should clang when they collide" — adjusting friction, restitution, and sound-on-impact properties - 78.8.8.4 Implement
PhysicsPreviewSimulatorrunning accelerated physics simulations in the preview pane to show the effects of physics changes across multiple scenarios (NPC walking, projectile flight, vehicle handling, ragdoll falls)
78.8.9 AI Behavior Modding Domain#
- 78.8.9.1 Create
AIBehaviorModHandlersupporting: behavior tree editing, utility function tuning, perception range adjustment, aggression/ fear thresholds, patrol route definition, group behavior rules, faction relationships - 78.8.9.2 Implement
BehaviorDescriptionCompiler— "wolves should hunt in packs of 3-5, circle their prey, and retreat if they lose half their pack" — compiling natural language into behavior tree nodes with conditions, actions, and decorators - 78.8.9.3 Create
AIDebugVisualizershowing NPC decision-making in real-time in the Glass Workshop — perception cones, threat assessment scores, behavior tree execution path, utility function evaluations - 78.8.9.4 Implement
AIBehaviorTesterthat spawns test NPCs in the preview sandbox and runs behavior scenarios — place a pack of wolves near a deer, observe the modded hunting behavior, iterate via voice
78.8.10 Audio Modding Domain#
- 78.8.10.1 Create
AudioModHandlersupporting: sound effect replacement, ambient soundscape design, music track assignment, spatial audio configuration, reverb zone definition, audio trigger placement - 78.8.10.2 Implement
SoundscapeDesigner— "this cave should echo with dripping water and distant rumbling" — combining procedural audio synthesis, sample selection, and spatial audio placement - 78.8.10.3 Create
MusicMoodMapper— "play tense music when enemies are nearby and calm music when exploring" — configuring adaptive music system triggers and crossfade parameters - 78.8.10.4 Implement
AudioPreviewMixerallowing isolated preview of audio changes — hear the new soundscape without other game audio, or hear it blended with the full mix
78.8.11 Economy Modding Domain#
- 78.8.11.1 Create
EconomyModHandlersupporting: price adjustment, drop rate modification, currency creation, shop inventory editing, trade rule definition, resource scarcity tuning, crafting recipe modification - 78.8.11.2 Implement
EconomySimulator— run 1000 simulated player- hours with modified economy parameters and report: average gold at each hour, item availability curves, inflation/deflation rates, gear progression speed - 78.8.11.3 Create
EconomyVisualizationPanelwith supply/demand curves, price history graphs, and wealth distribution charts updated in real-time as parameters are adjusted
78.8.12 Social Systems Modding Domain#
- 78.8.12.1 Create
SocialModHandlersupporting: faction relationship editing, reputation system tuning, trade rule modification, guild permission configuration, communication channel creation, emote/gesture creation - 78.8.12.2 Implement
FactionRelationshipEditor— "make the elves and dwarves hostile but willing to trade" — adjusting faction matrices, diplomatic state transitions, and trade permission rules - 78.8.12.3 Create
ReputationCurveDesigner— "it should be easy to gain reputation at first but very hard to reach max level" — defining non-linear reputation progression with visual curve editor
78.8.13 Accessibility Modding Domain#
- 78.8.13.1 Create
AccessibilityModHandlersupporting: font size/style adjustment, color scheme modification (colorblind modes), subtitle configuration, audio cue alternatives for visual events, input remapping, difficulty scaling, motion sensitivity options - 78.8.13.2 Implement
AccessibilityAutoAudit— automatically scan the current game state for accessibility issues: contrast ratios below WCAG AA, missing alt text, audio-only cues without visual alternatives, untimed-out quick-time events - 78.8.13.3 Create
AccessibilityPresetGenerator— "make the game playable for someone who is colorblind and hard of hearing" — applying a comprehensive set of accessibility modifications from a single voice command
78.9 Real-Time Generation Pipeline#
Live asset, code, and content generation with mid-generation voice feedback loop.
78.9.1 Generation Orchestration#
- 78.9.1.1 Create
libs/maya/forge-companion/generation-pipeline/library orchestrating all generative capabilities from Isis Factory and other domain services - 78.9.1.2 Implement
GenerationOrchestratormanaging concurrent generation requests with priority queuing, resource budgeting, and pipeline coordination - 78.9.1.3 Create
GenerationProgressStreamproviding real-time progress updates for every generation task — percentage complete, current stage name, estimated time remaining, preview snapshots - 78.9.1.4 Implement
MidGenerationFeedbackLoop— player can redirect ongoing generation via voice: "make the horns curlier", "use more red", "that's too big" — feedback is injected into the generation pipeline without restarting from scratch - 78.9.1.5 Create
GenerationQualityGateverifying generated assets meet minimum quality thresholds before presenting to player — polygon count within budget, texture resolution matches target, animation has no jitter, code passes lint - 78.9.1.6 Implement
GenerationCostEstimatorwarning players before expensive generation operations — "this will generate 50 unique trees, which will take about 2 minutes — want to proceed?" - 78.9.1.7 Create
GenerationCacheManagercaching previously generated assets by semantic description — "a red sword" returns the cached result if one exists, with "regenerate" option for fresh output
78.9.2 Text-to-3D Model Generation#
- 78.9.2.1 Create
TextTo3DGeneratorintegrating Isis 3D generation pipeline (Tripo AI, Meshy, or local models) with voice-driven prompting - 78.9.2.2 Implement
ModelProgressVisualizershowing the model being built in real-time in the Glass Workshop — shape emerging from noise, details refining, textures painting on - 78.9.2.3 Create
ModelRefinementLoop— after initial generation, player can request specific changes: "make it taller", "add more detail to the face", "smooth out the base" — applied as targeted refinements - 78.9.2.4 Implement
AutoRetopologyprocessing generated meshes into game-ready topology with configurable polygon budgets per LOD level - 78.9.2.5 Create
AutoRiggingfor generated characters/creatures — detect skeleton structure, place bones, paint weights and create a basic control rig. (Blocker made visible 2026-09-18: the prior HTML comment correctly said a bounding-box heuristic would be a stub and this needs an ML/geometric-inference model.) Use a published RigNet-class model for inference (install-first: fetch the pretrained weights + runtime, record versions, run on CPU; park only with a named failed install attempt) rather than a heuristic. Verify: a spec rigs a fixture humanoid mesh and asserts the predicted joints land within a stated distance of the reference skeleton and that skin weights are normalised per vertex; a non-humanoid mesh is handled or reported, not faked. - 78.9.2.6 Implement
PBRMaterialAutoAssignmentgenerating and assigning physically-based materials (albedo, roughness, metallic, normal, AO, emission) to generated meshes based on the text description - 78.9.2.7 Create
GeneratedAssetIntegrationthat automatically places generated 3D models into the game world with correct scale, physics collider, and LOD configuration - 78.9.2.8 Implement text-to-3D generation tests verifying mesh quality, PBR material correctness, rigging validity, and integration placement
78.9.3 Text-to-Texture & Material Generation#
- 78.9.3.1 Create
TextToTextureGeneratorproducing textures from voice descriptions — "mossy stone wall", "rusted metal grate", "alien bioluminescent skin" — generating tileable PBR texture sets - 78.9.3.2 Implement
TextureApplicationPreviewshowing the generated texture applied to the target mesh in real-time, with UV mapping visualization to catch stretching/seaming issues - 78.9.3.3 Create
MaterialPropertyEditor— after generation, fine-tune material properties via voice: "make it shinier", "more rough", "add a slight glow" — adjusting PBR parameters with live preview - 78.9.3.4 Implement
TextureVariationGeneratorproducing N variations of a texture from a single description — "give me 5 different stone wall options" — displayed as a selection grid in the Glass Workshop
78.9.4 Text-to-Animation Generation#
- 78.9.4.1 Create
TextToAnimationGeneratorproducing motion clips from voice descriptions — "a happy dance", "an intimidating battle cry", "a sneaky crouch walk" — using motion generation models (MotionDiffuse, MDM, or DeepMotion SayMotion integration) - 78.9.4.2 Implement
AnimationRetargeteradapting generated animations to any skeleton in the game — human animations retargeted to fantasy creatures, with automatic bone mapping and proportion adjustment - 78.9.4.3 Create
AnimationBlendTreeGenerator— "blend between walking and running based on speed" — automatically creating blend trees from voice descriptions with correct parameter bindings - 78.9.4.4 Implement
AnimationPreviewPanelin Glass Workshop showing the generated animation on the target character with play/pause/loop controls and speed adjustment
78.9.5 Text-to-Audio Generation#
- 78.9.5.1 Create
TextToSFXGeneratorproducing sound effects from voice descriptions — "a metallic clang", "a magical shimmer", "footsteps on gravel" — using Isis audio-generation pipeline - 78.9.5.2 Implement
TextToMusicGeneratorcreating music loops and tracks from voice descriptions — "epic orchestral battle music", "calm acoustic campfire music" — via Euterpe integration - 78.9.5.3 Create
TextToAmbientGeneratorproducing ambient soundscapes — "a busy medieval marketplace", "a quiet alien forest at night" — layering multiple generated audio elements - 78.9.5.4 Implement
AudioPreviewPlayerin Glass Workshop with waveform visualization, spatial preview (hear from avatar's position vs. source position), and A/B comparison between generated options
78.9.6 Text-to-Code Generation#
- 78.9.6.1 Create
TextToCodeGeneratorproducing game scripts from voice descriptions with support for Luau (Roblox-style), TypeScript (web), Lua (lightweight runtime), and WASM (high-performance) - 78.9.6.2 Implement
CodeSandboxRunnerexecuting generated code in an isolated WASM sandbox, capturing output, errors, and performance metrics, and presenting results in the Glass Workshop - 78.9.6.3 Create
CodeExplanationGenerator— generated code is accompanied by natural language explanations at the player's expertise level, displayed as inline comments or a separate explanation panel - 78.9.6.4 Implement
CodeIterationLoop— "that works but make it also handle the case where the player has a shield" — iterative refinement of generated code with diff tracking between iterations - 78.9.6.5 Create
APIDiscoveryAssistant— when a player describes functionality that requires engine APIs, the assistant discovers and suggests relevant APIs, explaining their usage and limitations
78.9.7 Text-to-Narrative Generation#
- 78.9.7.1 Create
TextToDialogueGeneratorproducing dialogue trees from voice descriptions — "a mysterious merchant who hints at a hidden dungeon" — generating multiple dialogue paths, emotion tags, and voice direction notes - 78.9.7.2 Implement
TextToQuestGeneratorcreating full quest structures from voice descriptions — "a quest where you help a farmer find their lost cow, but the cow turns out to be a shapeshifter" — generating objectives, waypoints, rewards, and branching outcomes - 78.9.7.3 Create
TextToLoreGeneratorproducing world lore entries, item descriptions, faction histories, and environmental storytelling elements from voice descriptions - 78.9.7.4 Implement
NarrativeConsistencyValidatorverifying generated narrative content is consistent with existing game lore, character relationships, and world rules — using Hathor's lore knowledge graph
78.9.8 Text-to-World Generation#
- 78.9.8.1 Create
TextToTerrainGeneratorproducing terrain patches from voice descriptions — "a volcanic island with a central crater lake" — using Maya Genesis terrain pipeline with erosion simulation - 78.9.8.2 Implement
TextToUrbanGeneratorcreating urban environments from voice descriptions — "a cyberpunk back alley with neon signs and steam vents" — using Maya genesis-urban pipeline - 78.9.8.3 Create
TextToFloraGeneratorpopulating environments with vegetation from voice descriptions — "a bioluminescent mushroom forest with towering fungi and floating spores" — using Maya genesis-flora with ecosystem simulation - 78.9.8.4 Implement
WorldGenerationProgressVisualizershowing terrain forming in real-time in the Glass Workshop — elevation rising, erosion carving, vegetation growing, buildings appearing
78.10 Safety, Sandboxing & Mod Integrity#
Comprehensive safety layer ensuring mods can't break games, exploit players, or violate content policies.
78.10.1 Mod Execution Sandbox#
- 78.10.1.1 Create
libs/maya/forge-companion/mod-sandbox/library providing isolated execution environments for all mod code - 78.10.1.2 Implement
WASMSandboxRuntimeexecuting mod scripts in isolated WASM containers with configurable capabilities — filesystem access (none), network access (none), API access (whitelisted game APIs only), memory limit, CPU time limit - 78.10.1.3 Create
SandboxCapabilitySystemwith fine-grained permission grants — mods declare required capabilities ("physics_read", "entity_spawn", "ui_create"), sandbox enforces these at runtime - 78.10.1.4 Implement
SandboxResourceMonitortracking per-mod CPU usage, memory allocation, draw call count, and physics body count in real-time, with soft warnings and hard limits - 78.10.1.5 Create
SandboxEscapeDetectormonitoring for sandbox boundary violations — attempts to access restricted APIs, memory corruption exploits, timing side channels - 78.10.1.6 Implement
ModIsolationensuring mods cannot affect each other's state — separate memory spaces, separate ECS component namespaces, no shared mutable state except through declared interfaces - 78.10.1.7 Create sandbox unit tests with adversarial mod code attempting 20+ exploit patterns (buffer overflow, infinite loop, memory bomb, API abuse, cross-mod state corruption)
78.10.2 Performance Budget System#
- 78.10.2.1 Create
PerformanceBudgetAllocatorassigning CPU, GPU, memory, and bandwidth budgets to each active mod based on platform capability and total mod count - 78.10.2.2 Implement
PerformanceBudgetEnforcerthrottling or suspending mods that exceed their budget — with player notification and avatar explanation of why - 78.10.2.3 Create
BudgetVisualizationin Glass Workshop showing per-mod resource consumption as stacked bar charts with budget lines - 78.10.2.4 Implement
AdaptiveBudgetScalingthat automatically reduces mod budgets on lower-end hardware to maintain target framerate
78.10.3 Content Safety & IP Protection#
- 78.10.3.1 Create
ContentPolicyEngineintegrating Themis content moderation — checks all generated text, images, audio, and 3D models against configurable content policies before delivery - 78.10.3.2 Implement
IPProtectionScannerintegrating Themis copyright shield — checks generated assets for similarity to known copyrighted content (character likenesses, logo shapes, music melodies, brand names) - 78.10.3.3 Create
VoiceContentFilterscreening voice input for abusive, harmful, or policy-violating commands before they reach the intent classifier - 78.10.3.4 Implement
GeneratedContentWatermarkembedding invisible watermarks in all AI-generated assets (images, audio, 3D models) for provenance tracking - 78.10.3.5 Create
ContentPolicyCustomizationallowing game creators to define their own content policies (age rating, violence level, sexual content, real-world references) that the AI assistant enforces
78.10.4 Multiplayer Mod Safety#
- 78.10.4.1 Create
MultiplayerModNegotiator— in multiplayer, proposed mods are announced to affected players who can accept, reject, or counter-propose via voice or UI - 78.10.4.2 Implement
ServerAuthoritativeModValidator— all mods in multiplayer are validated server-side before application, preventing client-side exploits - 78.10.4.3 Create
ModCompatibilityMatrix— tracks which mods are compatible in multiplayer, detects conflicts (Player A's gravity mod vs. Player B's gravity mod), and proposes resolutions - 78.10.4.4 Implement
AntiCheatModFilterdistinguishing between creative mods (cosmetic, content, experience) and unfair advantage mods (wallhack, aimbot, speed hack), blocking the latter in competitive contexts
78.11 Semantic Mod Merging#
Intent-aware conflict resolution replacing traditional load-order systems.
78.11.1 Semantic Merge Engine#
- 78.11.1.1 Create
libs/maya/forge-companion/semantic-merge/library implementing intent-aware mod conflict resolution - 78.11.1.2 Implement
ModIntentExtractoranalyzing each mod to determine its semantic intent — what is it trying to achieve, not just what values it changes (e.g., "make combat harder" vs. "set enemy health to 200") - 78.11.1.3 Create
ConflictDetectoridentifying conflicts at the semantic level — two mods modifying gravity aren't necessarily in conflict if one targets indoor zones and the other targets outdoor zones - 78.11.1.4 Implement
ConflictResolutionStrategies— 6 resolution modes: average (split the difference), prioritize (one wins), scope-split (each applies in different contexts), parameterize (expose a player-facing slider), compose (apply sequentially), and creative (AI proposes a novel solution that satisfies both intents) - 78.11.1.5 Create
ConflictResolutionUIpresenting conflicts in the Glass Workshop with side-by-side comparison, AI-recommended resolution, and player choice - 78.11.1.6 Implement
MergePreviewshowing the result of each resolution strategy in the preview pane so the player can evaluate before committing - 78.11.1.7 Create
ModDependencyGraphtracking which mods depend on which game systems and other mods, enabling safe removal and reordering - 78.11.1.8 Implement semantic merge unit tests with 30+ conflict scenarios across all 12 mod domains verifying correct intent extraction, conflict detection, and resolution strategy application
78.12 Social & Distribution Integration#
Integration with Maya Bazaar (marketplace), Maya Variants (forking), and Maya Agora (governance) for sharing and community.
78.12.1 Mod Sharing & Publishing#
- 78.12.1.1 Create
ModPublisherintegrating with Maya Bazaar — package a mod (all changes from the timeline) with metadata, description, screenshots, and preview video for marketplace publishing - 78.12.1.2 Implement
VoiceModDescription— the avatar interviews the player about their mod via voice and generates a compelling marketplace description, changelog, and installation guide from the conversation - 78.12.1.3 Create
ModPreviewGeneratorautomatically generating before/after screenshots/video from the timeline for marketplace listing - 78.12.1.4 Implement
ModDependencyPackagerresolving and bundling all mod dependencies (generated assets, scripts, configs) into a self- contained package with versioned references
78.12.2 Mod Forking & Remixing (Maya Variants)#
- 78.12.2.1 Create
ModForkingInterface— player says "I like this mod but want to change the colors" and the system creates a fork with full attribution to the original creator - 78.12.2.2 Implement
ForkDiffVisualizationshowing exactly what the fork changes relative to the original, maintaining a clear diff chain - 78.12.2.3 Create
UpstreamMergeNotifier— when the original mod updates, forks are notified and the AI can automatically merge upstream changes (or flag conflicts for player resolution) - 78.12.2.4 Implement
AttributionChainTrackermaintaining full provenance — "this mod was forked from X by Y, which was forked from Z by W" — displayed in marketplace listings
78.12.3 Community Governance (Maya Agora)#
- 78.12.3.1 Create
CommunityVoteInterface— communities can vote on which community mods become "canonical" (integrated into the base game experience) via Maya Agora integration - 78.12.3.2 Implement
ModCurationAI— the AI assistant helps community moderators evaluate submitted mods for quality, safety, and coherence with the existing experience - 78.12.3.3 Create
ModCollectionManager— curated collections of mods ("Best combat overhauls", "Immersive weather pack") shareable as one-click installs - 78.12.3.4 Implement
CreatorRevenueTrackerintegrating with Maya Bazaar's revenue sharing — creators see earnings, download stats, and usage analytics in the Glass Workshop
78.12.4 Collaborative Modding#
- 78.12.4.1 Create
CollaborativeModSessionenabling multiple players to mod together in real-time — each player has their own AI assistant, all working in a shared mod space with real-time sync - 78.12.4.2 Implement
ModSessionVoiceChannelwith spatial voice chat between collaborating modders, separate from the AI assistant voice channel - 78.12.4.3 Create
CollaborativeConflictResolverhandling simultaneous edits to the same system by different modders — AI mediates and suggests compatible approaches - 78.12.4.4 Implement
ModSessionReplayrecording a collaborative modding session (all voice, all workspace activity, all changes) as a shareable/watchable tutorial
78.13 Learning & Adaptation System#
AI assistant that gets smarter about each player over time.
78.13.1 Player Skill Profiling#
- 78.13.1.1 Create
libs/maya/forge-companion/learning-system/library managing player modeling, adaptation, and progressive complexity - 78.13.1.2 Implement
SkillProfilertracking player expertise across all 12 mod domains on a continuous 0-1 scale — updated based on vocabulary complexity, command granularity, error frequency, and clarification request rate - 78.13.1.3 Create
ExpertiseDetectoranalyzing speech patterns to classify expertise — novice uses lay terms ("make it look cool"), intermediate uses domain terms ("increase the specular"), expert uses technical terms ("adjust the GGX distribution alpha") - 78.13.1.4 Implement
AdaptiveComplexityControlleradjusting Glass Workshop detail level, avatar explanation depth, confirmation protocol strictness, and proactive suggestion frequency based on skill profile - 78.13.1.5 Create
SkillProgressionTrackerdetecting when players are growing in expertise and gradually introducing more advanced concepts — "you've been adjusting materials a lot — want me to show you how the shader graph works?"
78.13.2 Preference Learning#
- 78.13.2.1 Implement
AestheticPreferenceModellearning the player's visual taste from their choices — color palettes, material preferences, environmental moods, architectural styles — used to bias generation toward preferred aesthetics - 78.13.2.2 Create
WorkflowPatternRecognizerdetecting the player's common modding patterns — "you usually adjust physics first, then test in the arena, then tweak AI behavior" — proactively preparing relevant panels and tools - 78.13.2.3 Implement
FeedbackSignalExtractorlearning from implicit feedback — when the player says "perfect" or "that's what I wanted", the system records what worked; when they say "no" or "undo", it records what didn't — building a preference model without explicit feedback - 78.13.2.4 Create
PersonalShortcutLearnerdetecting frequently used multi-step operations and offering to create voice shortcuts — "you always adjust gravity then test jumping — want me to make that one command?"
78.13.3 Teaching & Challenge Modes#
- 78.13.3.1 Create
TeachingMode— when the player is learning, the avatar slows down, explains more, shows more reasoning, and offers guided tutorials: "Want me to walk you through how the physics system works?" - 78.13.3.2 Implement
GuidedModTutorials— structured tutorial sequences where the avatar guides the player through increasingly complex modding tasks with scaffolded support - 78.13.3.3 Create
ChallengeMode— for expert players, the avatar suggests advanced techniques: "Did you know you can chain behavior trees with custom blackboard variables? Want me to show you?" - 78.13.3.4 Implement
ModInspirationEngine— when the player seems stuck or idle, the avatar proactively suggests interesting mods based on the current game context: "This area could really use some ambient particle effects — want to try adding fireflies?" - 78.13.3.5 Create
CrossSessionContinuity— the assistant remembers the exact state of the last session and offers to resume: "Last time we were working on the weather system — the rain was looking great but we hadn't done lightning yet. Pick up where we left off?"
78.14 Integration Orchestration Layer#
The central orchestration layer fusing all systems into a unified experience.
78.14.1 Companion Orchestrator#
- 78.14.1.1 Create
libs/maya/forge-companion/orchestrator/library as the central nervous system connecting voice, avatar, workshop, mod engine, generation, safety, social, and learning systems - 78.14.1.2 Implement
CompanionOrchestratormain class managing the full event loop: capture voice → ASR → intent → LLM reasoning → mod execution → workshop update → TTS → avatar animation → preview update, with streaming at every stage - 78.14.1.3 Create
EventBusconnecting all subsystems with typed events: VoiceInputEvent, IntentEvent, ModOperationEvent, GenerationEvent, WorkshopUpdateEvent, AvatarAnimationEvent, SafetyCheckEvent - 78.14.1.4 Implement
PipelineLatencyMonitortracking end-to-end latency from voice input to visible response, with per-stage breakdowns and alerting when latency exceeds 500ms (voice-to-response) or 850ms (voice-to-action) - 78.14.1.5 Create
GracefulDegradationController— when system load is high, gracefully reduce quality: fewer avatar animation frames, simpler workshop transitions, lower-res previews, rather than increasing latency - 78.14.1.6 Implement
SessionStateManagerpersisting the full companion state (conversation, timeline, workshop layout, mod state, avatar emotion) across save/load and connection interruptions
78.14.2 LLM Tool Calling Integration#
- 78.14.2.1 Create
ModToolDefinitions— define every mod operation as an LLM tool/function call with typed parameters, descriptions, and examples for the orchestrating LLM - 78.14.2.2 Implement
ToolCallExecutortranslating LLM tool calls to mod abstraction layer operations with error handling, rollback, and result formatting - 78.14.2.3 Create
ToolResultFormatterproducing human-readable descriptions of tool call results for TTS narration and Glass Workshop display - 78.14.2.4 Implement
ParallelToolExecutionsupporting concurrent tool calls when the LLM determines operations are independent — e.g., changing sky color and adjusting NPC behavior simultaneously - 78.14.2.5 Create
ToolCallAuditLogrecording every tool call with arguments, results, and timestamps for timeline reconstruction and debugging
78.14.3 Engine Adapter Layer#
- 78.14.3.1 Create
EngineAdapterInterfacedefining the abstract interface between the forge companion and any game engine — implemented by Bellona adapters for Godot, Unreal, Unity, and Maya Engine Core - 78.14.3.2 Implement
MayaEngineAdapterconnecting the forge companion to Maya Engine Core's ECS, rendering, physics, audio, and scripting systems - 78.14.3.3 Create
GodotEngineAdapterconnecting the forge companion to Godot's scene tree, GDScript, and engine APIs via GDExtension - 78.14.3.4 Implement
UnrealEngineAdapterconnecting the forge companion to Unreal's Blueprint system, C++ APIs, and editor tools via UE plugin - 78.14.3.5 Create
UnityEngineAdapterconnecting the forge companion to Unity's Component system, C# APIs, and editor extensions - 78.14.3.6 Implement
EngineCapabilityDiscovery— the adapter automatically reports which modding capabilities the connected engine supports, so the AI assistant only offers available operations
78.15 End-to-End Testing & Quality Assurance#
Comprehensive testing across all systems ensuring production-quality experience.
78.15.1 Voice Pipeline E2E Tests#
- 78.15.1.1 Create end-to-end voice pipeline tests with synthetic speech input → ASR → intent → LLM → TTS → audio output, verifying correctness and latency across 50+ mod scenarios
- 78.15.1.2 Implement noise robustness tests — run the full pipeline with game audio at 10 different noise levels (SNR from -5dB to 30dB) and verify >90% intent accuracy at SNR >5dB
- 78.15.1.3 Create multi-turn conversation tests — 20 scripted 10-turn conversations covering topic changes, interruptions, corrections, spatial references, and compound commands
- 78.15.1.4 Implement latency regression tests — continuous benchmark asserting voice-to-response <500ms and voice-to-action <850ms at p95
78.15.2 Avatar Animation E2E Tests#
- 78.15.2.1 Create avatar expression accuracy tests — feed emotional states and verify correct FACS AU activations, lip sync viseme sequences, and gesture timing
- 78.15.2.2 Implement avatar smoothness tests — verify no animation pops, jitter, or discontinuities during emotion transitions, interruptions, and rapid state changes
- 78.15.2.3 Create avatar personality consistency tests — verify that each personality preset produces measurably different behaviors across voice, gesture, expression, and dialogue
- 78.15.2.4 Implement avatar performance tests — verify avatar rendering and animation stay within a CPU/GPU budget. (Blocker made visible 2026-09-18: the prior HTML comment said this needs profiling on target hardware.) Measure on the fleet's desktop targets: CPU on the Linux server and GPU on the Mac (Metal), reporting per-frame CPU time and GPU time for a fixture avatar over 300 frames. Verify: the desktop budget is asserted with the measured numbers recorded; mobile and console budgets are a hardware residual (no phone/console in the fleet).
78.15.3 Glass Workshop E2E Tests#
- 78.15.3.1 Create workshop layout tests — verify panels arrange correctly, don't overlap, and maintain readability across screen sizes and VR/flat modes
- 78.15.3.2 Implement workshop interaction tests — verify voice-driven panel commands ("show me the code", "hide the workshop") work correctly with <200ms response
- 78.15.3.3 Create timeline rollback tests — apply 20 mods, then roll back to each point in the timeline, verifying exact game state restoration
- 78.15.3.4 Implement impact-assessment accuracy tests — compare the
predicted mod impact with the actual impact across 30+ mod operations,
verifying > 85% prediction accuracy. (Blocker made visible 2026-09-18: the
prior HTML comment said measuring actual impact needs applying mods in a
running game and observing real cross-system effects.) Needs a labelled
corpus of 30+ real mod operations with their observed in-game impact to
score prediction accuracy against, which does not exist (the predictor's
internal ripple-consistency is already tested in the workshop E2E suite)
(stated 2026-09-18).
blocked:corpus
78.15.4 Mod Engine E2E Tests#
- 78.15.4.1 Create mod abstraction layer tests — 100+ natural language mod commands across all 12 domains, verifying correct translation to engine primitives and correct game state changes
- 78.15.4.2 Implement mod safety tests — adversarial mod commands attempting to break game state, crash the engine, or exploit sandbox boundaries, verifying all are caught and rejected
- 78.15.4.3 Create semantic merge tests — 30+ mod conflict scenarios verifying correct conflict detection, resolution strategy selection, and merged result correctness
- 78.15.4.4 Implement generation pipeline tests — text-to-3D, text-to- texture, text-to-animation, text-to-code, text-to-narrative generation with quality gate validation
78.15.5 Integration & Stress Tests#
- 78.15.5.1 Create full-system integration test — a complete modding session from voice input through generation, preview, approval, and application, covering all subsystems end-to-end
- 78.15.5.2 Implement concurrent modding stress test — 10 simultaneous mod operations with generation, preview, and application, verifying no deadlocks, race conditions, or state corruption
- 78.15.5.3 Create multi-session continuity test — start session, make mods, close, reopen, verify all state is preserved and assistant resumes correctly
- 78.15.5.4 Implement multiplayer modding test — two players modding the same game simultaneously with conflict resolution and mod negotiation
- 78.15.5.5 Create cross-engine compatibility test — run the same modding scenario against Maya Engine, Godot, Unreal, and Unity adapters, verifying functional parity
Phase 78 Summary — Maya Forge AI Modding Companion#
New Libraries:
| Library | Section | Tasks |
|---|---|---|
| Voice ASR: Core Pipeline | 78.1.1 | 15 |
| Voice ASR: Spatial Commands | 78.1.2 | 10 |
| Voice ASR: Intent & Entity | 78.1.3 | 12 |
| Voice TTS: Streaming Pipeline | 78.2.1 | 13 |
| Voice TTS: Dialogue Management | 78.2.2 | 10 |
| Avatar: FACS Blend Shapes | 78.3.1 | 10 |
| Avatar: Lip Sync | 78.3.2 | 8 |
| Avatar: Gesture Generation | 78.4.1 | 8 |
| Avatar: Idle & Ambient | 78.4.2 | 6 |
| Avatar: Emotional State Machine | 78.4.3 | 6 |
| Avatar: Personality Framework | 78.5.1 | 7 |
| Avatar: Appearance Customization | 78.5.2 | 6 |
| Workshop: Spatial UI Framework | 78.6.1 | 12 |
| Workshop: Live Diff Viewer | 78.6.2 | 6 |
| Workshop: System Graph | 78.6.3 | 5 |
| Workshop: Reasoning Stream | 78.6.4 | 5 |
| Workshop: Timeline & Rollback | 78.7.1 | 8 |
| Workshop: Live Preview | 78.7.2 | 6 |
| Workshop: Impact Assessment | 78.7.3 | 8 |
| Workshop: Confidence & Progress | 78.7.4 | 4 |
| Mod Engine: Abstraction Stack | 78.8.1 | 10 |
| Mod Engine: Asset Domain | 78.8.2 | 5 |
| Mod Engine: Code Domain | 78.8.3 | 5 |
| Mod Engine: Rules Domain | 78.8.4 | 4 |
| Mod Engine: World Domain | 78.8.5 | 5 |
| Mod Engine: UI Domain | 78.8.6 | 4 |
| Mod Engine: Narrative Domain | 78.8.7 | 5 |
| Mod Engine: Physics Domain | 78.8.8 | 4 |
| Mod Engine: AI Behavior Domain | 78.8.9 | 4 |
| Mod Engine: Audio Domain | 78.8.10 | 4 |
| Mod Engine: Economy Domain | 78.8.11 | 3 |
| Mod Engine: Social Domain | 78.8.12 | 3 |
| Mod Engine: Accessibility Domain | 78.8.13 | 3 |
| Generation: Orchestration | 78.9.1 | 7 |
| Generation: Text-to-3D | 78.9.2 | 8 |
| Generation: Text-to-Texture | 78.9.3 | 4 |
| Generation: Text-to-Animation | 78.9.4 | 4 |
| Generation: Text-to-Audio | 78.9.5 | 4 |
| Generation: Text-to-Code | 78.9.6 | 5 |
| Generation: Text-to-Narrative | 78.9.7 | 4 |
| Generation: Text-to-World | 78.9.8 | 4 |
| Safety: Mod Sandbox | 78.10.1 | 7 |
| Safety: Performance Budget | 78.10.2 | 4 |
| Safety: Content & IP | 78.10.3 | 5 |
| Safety: Multiplayer | 78.10.4 | 4 |
| Semantic Merge Engine | 78.11.1 | 8 |
| Social: Sharing & Publishing | 78.12.1 | 4 |
| Social: Forking & Remixing | 78.12.2 | 4 |
| Social: Community Governance | 78.12.3 | 4 |
| Social: Collaborative Modding | 78.12.4 | 4 |
| Learning: Skill Profiling | 78.13.1 | 5 |
| Learning: Preference Learning | 78.13.2 | 4 |
| Learning: Teaching & Challenge | 78.13.3 | 5 |
| Orchestration: Core | 78.14.1 | 6 |
| Orchestration: LLM Tools | 78.14.2 | 5 |
| Orchestration: Engine Adapters | 78.14.3 | 6 |
| Testing: Voice E2E | 78.15.1 | 4 |
| Testing: Avatar E2E | 78.15.2 | 4 |
| Testing: Workshop E2E | 78.15.3 | 4 |
| Testing: Mod Engine E2E | 78.15.4 | 4 |
| Testing: Integration & Stress | 78.15.5 | 5 |
| TOTAL Phase 78 | 355 |
Industry SOTA Coverage After Phase 78#
| Capability | Before Phase 78 | After Phase 78 | Industry Leaders |
|---|---|---|---|
| Voice-first modding | None | Full natural language voice modding across 12 domains | NO COMPETITOR — first in industry |
| Embodied AI companion | None | FACS-animated avatar with personality, emotion, gesture, lip sync | NO COMPETITOR — Roblox has text-only chat |
| Spatial voice commands | None | Gaze + gesture + raycast spatial resolution for deictic references | NO COMPETITOR — no modding tool has spatial voice |
| Transparent AI workspace | None | Glass Workshop with diffs, graphs, reasoning, timeline, preview | EXCEEDS Devin/Cursor (which are code-only, not game modding) |
| All-granularity modding | Load-order conflicts, expertise-gated | 4-layer abstraction (natural language → engine primitives) | EXCEEDS Bethesda (deep but inaccessible) and Roblox (accessible but shallow) |
| Semantic mod merging | None (load-order only) | Intent-aware conflict resolution with 6 strategies | NO COMPETITOR — all others use load-order |
| Live generation with feedback | Batch generation only | Mid-generation voice redirection for 3D, texture, animation, audio, code | NO COMPETITOR — no tool supports mid-gen feedback |
| AI mod safety | Manual review (CurseForge) | Real-time sandbox + content policy + IP scan + performance budget | EXCEEDS CurseForge, mod.io (which are manual/batch) |
| Collaborative modding | None | Multi-player real-time co-modding with AI mediation | NO COMPETITOR — no modding tool has collaborative AI |
| Mod forking/provenance | None | Git-like mod forking with full attribution chain | EXCEEDS Steam Workshop, CurseForge |
| Player skill adaptation | None | Continuous expertise profiling with adaptive complexity | NO COMPETITOR — no modding tool learns from user |
| Cross-engine modding | Engine-specific only | One interface for Maya, Godot, Unreal, Unity | NO COMPETITOR — all tools are engine-locked |
| Avatar personality | None | Big Five personality model with evolution and customization | EXCEEDS NVIDIA ACE (which has personalities but no modding context) |
| Mod impact assessment | None | Cross-system ripple analysis with performance, balance, narrative, accessibility checks | NO COMPETITOR — no tool analyzes mod impact holistically |
No single competitor achieves any ONE of these capabilities in a modding context, let alone all of them combined:
- Roblox Studio has AI text chat but no voice, no avatar, no Glass Workshop, no semantic merge, no mid-generation feedback, no cross-engine support
- Bethesda Creation Kit has deep modding but no AI, no voice, no avatar, no accessibility, requires expert knowledge
- Unity Muse has AI text generation but no voice, no avatar, no modding framework, no safety layer, no social features
- Unreal UEFN has visual scripting but no AI companion, no voice, no generation pipeline, no semantic merge
- Dreams had accessible creation but no AI, no voice, no generation, no social/distribution, and is no longer developed
- NVIDIA ACE has avatar tech but no modding context, no workshop, no generation, no semantic merge
- Devin/Cursor show AI work transparently but are code-only, not game modding, no avatar, no voice, no generation
Maya Forge AI Modding Companion after Phase 78 = The only system in the industry where a player can say "make the wolves hunt in packs and give them glowing eyes" and watch an expressive AI avatar physically generate the behavior tree, create the glow shader, preview the result in a transparent workspace, explain the cross-system impact, and apply it — all through natural conversation.
"In Maya's forge, every player is a creator — their voice is the hammer, their imagination is the anvil, and the AI companion is the master smith who brings their vision to life."