Historical task source. The Eve task board holds current task state.
Phase 82: Voice-as-First-Class Gameplay Input — Tactical Commands, NPC Dialogue & Squad Control#
Purpose: Make the player's voice a first-class input mechanism for gameplay — not just modding (Phase 78). Players can talk to any NPC in natural language, command tactical squads with real military voice procedures (Rainbow Six / SOCOM style), negotiate with faction leaders, interrogate prisoners, and direct AI teammates in real-time combat using voice. This phase combines on-device speech recognition, dual-path intent classification (fast BERT + fallback SLM), spatial reasoning from voice and gaze, voice-driven NPC dialogue, tactical squad command execution, multi-agent formation control, and personality-matched TTS feedback.
Reference Proposal: docs/proposals/LIVING_WORLD_NPC_SOTA_PROPOSAL.md
Scope: Rust crates for ASR inference, intent classification, spatial resolvers, and formation control. TypeScript libraries for voice dialogue orchestration, negotiation system, command taxonomy, and TTS pipeline. Integration with Phase 78 voice infrastructure (shared ASR), Phase 81 cognitive core (NPC reasoning), Phase 43 behavior trees (action execution), Phase 79.24 personality (personality-modulated responses), and Phase 79.25 relationships (history-aware dialogue).
Technology:
- Rust crates (in
libs/maya/engine-core/crates/):maya-voice-input(game-side ASR integration, VAD, wake word),maya-intent(BERT-FID ONNX inference),maya-spatial-voice(gaze + entity resolution),maya-formation(ORCA + Boids tactical formations),maya-squad-ai(tactical behavior trees) - TypeScript libraries (in
libs/maya/):voice-dialogue(NPC conversation orchestration),voice-negotiation(persuasion/intimidation/deception),voice-tactics(command taxonomy + decomposition),voice-tts(personality TTS pipeline) - External dependencies: whisper.cpp/CarelessWhisper (ASR), Picovoice Porcupine (wake word), ONNX Runtime (BERT-FID), Piper TTS (voice synthesis)
- Integration targets: Phase 78 (shared ASR infrastructure), Phase 81 (cognitive core for NPC reasoning), Phase 43 (behavior trees), Phase 79.24 (personality/emotions/memory), Phase 79.25 (relationships), Phase 79.31 (NPC-NPC interaction), Phase 79.34 (information propagation for gossip)
82.1 On-Device ASR Pipeline for Gameplay (Rust)#
Real-time speech recognition optimized for gameplay environments — noisy (combat SFX, music, ambient), low-latency (<200ms), and streaming (partial results while speaking). Shares infrastructure with Phase 78 ASR but adds gameplay-specific noise filtering and context routing.
82.1.1 Core ASR Engine#
- 82.1.1.1 Create
maya-voice-inputRust crate with safe FFI wrapper around whisper.cpp (CarelessWhisper variant for streaming, 300ms chunk size, causal encoder) - 82.1.1.2 Implement audio capture integration — receive microphone PCM data from platform audio subsystem (WASAPI/CoreAudio/ALSA), 16kHz mono, with circular buffer for continuous input
- 82.1.1.3 Create game audio cancellation — subtract game audio output (SFX, music, voice chat) from microphone input using adaptive echo cancellation to isolate player voice from speakers/headset bleed
- 82.1.1.4 Implement voice activity detection (VAD) — Silero VAD model (<2MB, <10ms latency) detecting speech onset within 50ms and speech end within 200ms, with hysteresis to prevent false triggers from game audio transients (explosions, gunfire)
- 82.1.1.5 Create noise-robust preprocessing — spectral gating to suppress game ambient noise, adaptive noise floor estimation that adjusts to current game audio environment (quiet exploration vs loud combat)
- 82.1.1.6 Implement streaming transcription — emit partial transcript within 150ms of speech onset, update every 100ms as more audio arrives, finalize within 200ms of speech end
- 82.1.1.7 Create utterance boundary detection — segment continuous speech into complete commands/sentences using pause detection (300ms default), prosodic falling intonation, and syntactic completeness estimation
- 82.1.1.8 Implement multi-language support — load Whisper multilingual model, detect language automatically within first 2 seconds of speech, support mid-conversation language switching
- 82.1.1.9 Create per-word confidence scoring — annotate each transcribed word with confidence (0-1), flag low-confidence words (<0.6) for potential clarification requests
- 82.1.1.10 Implement ASR model configuration — support multiple model sizes (tiny.en 75MB for command-only, medium 1.5GB for full conversation) with automatic selection based on available resources
- 82.1.1.11 Create ASR performance profiling — measure per-utterance latency, word error rate on gameplay audio test set, CPU/GPU usage
- 82.1.1.12 Implement ASR integration tests — test with recorded gameplay audio at various noise levels (quiet, combat, music) with ground truth transcriptions
82.2 Tactical Intent Classification — Fast Path (Rust)#
BERT-FID-inspired fast intent classifier for structured tactical commands. Achieves <100ms classification with high accuracy on the 25-category tactical command taxonomy. High-confidence results execute directly without SLM.
82.2.1 Intent Classifier#
- 82.2.1.1 Create
maya-intentRust crate with ONNX Runtime integration for BERT-class model inference - 82.2.1.2 Implement tokenizer — BPE tokenizer compatible with fine-tuned BERT model, max 64 tokens per utterance (tactical commands are short)
- 82.2.1.3 Create 3-layer hierarchical intent taxonomy with 25 categories: Layer 1 (team selection: alpha/bravo/charlie/everyone/nearest), Layer 2 (action: move/hold/stack/breach/engage/suppress/cease_fire/flash_clear/ smoke/cover/overwatch/formation/posture), Layer 3 (target/location extraction as named entities)
- 82.2.1.4 Implement named entity recognition — extract team names, location references ("that door", "left side", "second floor"), target references ("the hostile", "the vehicle"), and formation names ("wedge", "column", "diamond") as typed entities
- 82.2.1.5 Create confidence scoring with entropy optimization — compute output entropy, classify as high-confidence (<0.3 entropy → execute directly) or low-confidence (≥0.3 entropy → route to SLM planner §82.3)
- 82.2.1.6 Implement fast-path execution — high-confidence classified intents bypass SLM entirely and map directly to atomic action structs submitted to the squad AI (§82.7)
- 82.2.1.7 Create training data generation — use SLM to generate 50,000+ synthetic tactical command utterances with intent labels, augmented with paraphrasing and noise injection
- 82.2.1.8 Implement model fine-tuning pipeline — fine-tune BERT-base (110M params) on tactical command dataset, export to ONNX format, quantize to INT8 for inference
- 82.2.1.9 Create domain vocabulary injection — add game-specific terms (weapon names, location names, callsign names) to tokenizer vocabulary and retrain intent classifier when game content changes
- 82.2.1.10 Implement classification latency benchmark — verify <100ms end-to-end from text input to intent struct output on target hardware
- 82.2.1.11 Create intent classification evaluation suite — precision, recall, F1 per intent category, confusion matrix, with particular focus on dangerous misclassifications (e.g., "hold fire" classified as "fire")
- 82.2.1.12 Implement continuous learning — misclassified commands (detected when player repeats or corrects) logged for retraining
82.3 Natural Language Command Decomposition — SLM Path (TypeScript + Rust)#
For complex or ambiguous commands that the BERT classifier cannot resolve, a small language model decomposes natural language into sequences of atomic squad actions with temporal ordering, dependencies, and spatial resolution.
82.3.1 SLM Command Planner#
- 82.3.1.1 Create
libs/maya/voice-tactics/TypeScript library with command decomposition, temporal ordering, and dependency resolution - 82.3.1.2 Implement command prompt template — system prompt defining available atomic actions (move_to, breach, hold, engage, suppress, formation, etc.) with parameter schemas, plus current squad state (positions, ammo, health, readiness)
- 82.3.1.3 Create atomic action library — 40+ typed actions with
parameters:
move_to(agent, position),breach(agents[], door, method),suppress(agent, area, duration),formation(agents[], type, center),hold_position(agent, position, facing), etc. - 82.3.1.4 Implement multi-step decomposition — complex commands like "Alpha go around back and breach the kitchen while Bravo holds the front" decomposed into: [assign Alpha team, pathfind to rear, hold at kitchen door, assign Bravo team, hold at front door, synchronize breach on signal]
- 82.3.1.5 Create temporal dependency resolver — actions tagged with prerequisites (breach_after:signal, suppress_until:team_in_position), compiled into an execution DAG
- 82.3.1.6 Implement "on my mark" / signal system — commands with "on my mark" create a pending action set that executes atomically when player says "go" / "execute" / "breach"
- 82.3.1.7 Create conditional commands — "if they surrender, restrain them; if they resist, engage" → conditional branches in execution plan
- 82.3.1.8 Implement standing order system — "hold this position until I say otherwise" creates a persistent order that remains active until explicitly countermanded or the NPC is forced to break it (self- preservation)
- 82.3.1.9 Create command validation — verify decomposed plan is feasible given current squad state (enough ammo, path exists, agents alive), report impossibility to player via TTS if not
- 82.3.1.10 Implement plan preview — before execution, briefly flash squad member waypoints and action icons on HUD so player can verify the system understood correctly (optional, togglable)
82.4 Spatial Reasoning from Voice (Rust)#
Resolves spatial references in voice commands ("that door", "left flank", "the red building") to actual world coordinates using player gaze direction, proximity, entity metadata, and CLIP-based visual similarity.
82.4.1 Spatial Resolver#
- 82.4.1.1 Create
maya-spatial-voiceRust crate with spatial reference resolution pipeline - 82.4.1.2 Implement gaze-anchored reference resolution — "that" + entity_type resolves to the entity nearest to the player's crosshair raycast intersection, within 50m range, matching the specified type (door, window, vehicle, hostile, position)
- 82.4.1.3 Create relative spatial resolution — "left" = 90° left of squad facing direction in world space, "right" = 90° right, "behind" = 180° from movement vector, "ahead" = forward, "above" = +Y
- 82.4.1.4 Implement ordinal spatial references — "second door on the left" → enumerate all doors within FOV, sort left-to-right from player perspective, select index 2
- 82.4.1.5 Create named landmark resolution — "the church" / "the red building" → search entity database for matching name or description within 200m radius, rank by distance and visibility
- 82.4.1.6 Implement CLIP-based visual reference — "the building with the broken window" → run CLIP similarity between description embedding and visible entity visual feature embeddings, return highest match above threshold
- 82.4.1.7 Create contextual reference resolution — "the same spot" → retrieve from squad episodic memory the last referenced position; "where we entered" → retrieve entry point from mission memory
- 82.4.1.8 Implement proximity-based disambiguation — when multiple entities match a reference, prefer the one closest to the player or closest to the referenced team's current position
- 82.4.1.9 Create cover point inference — "take cover" → query navmesh cover point graph for nearest cover relative to threat direction, suitable for the specified agent
- 82.4.1.10 Implement spatial reference confidence — each resolution includes confidence score; low confidence triggers clarification: "Did you mean the north door or the south door?" via TTS
- 82.4.1.11 Create spatial resolution debugging — visualize resolved positions as markers in the game world, show alternative candidates and their scores
- 82.4.1.12 Implement spatial resolution benchmark — test suite with 100+ reference utterances in various tactical scenarios, measure resolution accuracy and disambiguation quality
82.5 Conversational NPC Interaction — Voice Dialogue Mode (TypeScript)#
Any Tier 0 NPC (Phase 81) can engage in natural voice conversation with the player. The system manages turn-taking, emotional responses, personality- consistent dialogue, and consequential actions from conversation.
82.5.1 Voice Dialogue System#
- 82.5.1.1 Create
libs/maya/voice-dialogue/TypeScript library with dialogue session management, turn-taking, and consequence engine - 82.5.1.2 Implement conversation initiation — detect when player is addressing an NPC via: proximity (<3m) + gaze (looking at NPC) + speech (VAD active) + optional wake word ("Hey [name]"). Activate dialogue mode for that NPC.
- 82.5.1.3 Create dialogue session state — track conversation history, current topic, NPC emotional trajectory, and player relationship changes accumulating during this conversation
- 82.5.1.4 Implement NPC response generation — player utterance fed to NPC's SLM (Tier 0 via Phase 81) with full context: personality, current emotion, relationship with player, memories of player, current activity, location context, knowledge/rumors, and conversation history
- 82.5.1.5 Create personality-consistent dialogue — SLM system prompt enforces personality-driven speech patterns: high Intelligence → complex vocabulary, high Humor → jokes/wit, low Education → simpler language, high Formality → respectful forms of address
- 82.5.1.6 Implement emotion-driven responses — NPC emotional state (Phase 79.24.3) modulates SLM response: frightened NPC speaks haltingly, angry NPC is curt and confrontational, happy NPC is warm and generous
- 82.5.1.7 Create knowledge-bounded responses — NPC only knows what they've experienced (memory stream) and what they've heard (gossip via Phase 79.34). NPC cannot reveal information they don't have.
- 82.5.1.8 Implement turn-taking management — detect when player stops speaking (VAD + pause), begin NPC response generation immediately, handle player interruption gracefully (NPC pauses or adjusts response)
- 82.5.1.9 Create conversation consequences — actions NPC decides to take based on conversation: give item, reveal information, change opinion of player, agree to quest, refuse request, offer trade, attack, flee. Consequences dispatch to game systems.
- 82.5.1.10 Implement conversation memory — entire exchange stored as episodic memories for both the NPC and the player profile, affecting future interactions
- 82.5.1.11 Create conversation ending — NPC naturally concludes conversation when topic exhausted, they need to resume activity, or emotional state makes continued conversation unlikely (anger → walks away, fear → runs)
- 82.5.1.12 Implement eavesdropping — nearby NPCs (Tier 0/1) can "overhear" player-NPC conversations, storing overheard information in their memory streams, potentially spreading it as gossip (Phase 79.34)
- 82.5.1.13 Create multi-NPC conversation — player can address a group of NPCs, each responds in turn based on personality (dominant personalities speak first, shy personalities wait), with NPCs reacting to each other's responses
- 82.5.1.14 Implement conversation logging — full transcripts stored for debugging, with NPC internal state (emotion, opinion changes) annotated at each turn
- 82.5.1.15 Create dialogue integration tests — scripted conversation scenarios verifying personality consistency, knowledge boundaries, and correct consequence dispatch
82.6 Voice Negotiation & Persuasion System (TypeScript)#
Players can negotiate, persuade, intimidate, and deceive NPCs through natural conversation. Outcomes depend on NPC personality, relationship, evidence, and the quality of the player's argument.
82.6.1 Negotiation Engine#
- 82.6.1.1 Create
libs/maya/voice-negotiation/TypeScript library with negotiation state machine, persuasion model, and outcome resolver - 82.6.1.2 Implement negotiation detection — identify when a conversation transitions from chat to negotiation (player makes a request, offer, or demand) and activate the negotiation engine
- 82.6.1.3 Create persuasion scoring model — SLM evaluates player argument quality considering: logical coherence, emotional appeal, evidence cited, reputation leverage, and alignment with NPC values
- 82.6.1.4 Implement personality-modulated resistance — NPC personality determines negotiation difficulty: high Stubbornness increases resistance, high Agreeableness decreases it, high Greed makes financial arguments more effective, high Honor makes duty arguments more effective
- 82.6.1.5 Create trade negotiation — "Can you do 30 gold instead of 50?" → NPC evaluates: item value, own need for gold, player relationship, surplus stock, personality (greedy vs generous). Returns counter-offer, acceptance, or firm refusal.
- 82.6.1.6 Implement diplomatic persuasion — "We need your militia against the bandits" → NPC evaluates: threat assessment, bravery, community values, self-interest, military strength. May impose conditions ("Only if you deal with the river crossing first").
- 82.6.1.7 Create intimidation mechanics — "You don't want to make an enemy of me" → NPC evaluates: player's reputation and apparent power vs own courage, available backup, consequences of compliance. Cowardly NPCs yield, brave NPCs call bluff.
- 82.6.1.8 Implement deception detection — "I'm a merchant guild inspector" → NPC evaluates: player appearance, known facts, NPC suspicion trait, whether claim is verifiable. High-Intelligence NPCs are harder to deceive.
- 82.6.1.9 Create multi-round negotiation — negotiation can span multiple exchanges with escalating or de-escalating tension, NPC patience tracking (eventually walks away if negotiation drags)
- 82.6.1.10 Implement witness effects — NPCs nearby who witness intimidation or deception may change their opinion of the player, or report behavior to authorities (Phase 79.29 justice system)
- 82.6.1.11 Create faction-level negotiation — player negotiating with a faction leader affects faction-wide disposition, potentially triggering Phase 79.36 political consequences
- 82.6.1.12 Implement negotiation memory — NPCs remember past negotiations, broken promises increase resistance in future dealings, kept promises increase trust
82.7 Squad Tactical Command Execution (Rust + TypeScript)#
Bridges voice commands to squad NPC behavior trees. Handles command queuing, synchronization points, and team assignment for tactical operations.
82.7.1 Command Execution Engine#
- 82.7.1.1 Create
maya-squad-aiRust crate with tactical behavior tree extensions, command queue, and synchronization primitives - 82.7.1.2 Implement command queue — per-team FIFO queue of pending commands. New voice commands append to queue. "Cancel" clears queue. "Override" replaces current execution.
- 82.7.1.3 Create team assignment system — maintain named teams (Alpha, Bravo, Charlie) with persistent membership. "You" targets nearest team member. "Everyone" targets all. Team composition changeable via voice.
- 82.7.1.4 Implement breach execution — coordinated multi-step: stack formation at breach point → breach method (kick/explosive/flashbang) → entry sequence (first/second/third person) → room clear pattern (left/ right split, center dominate, cross-entry)
- 82.7.1.5 Create hold and synchronize — "Breach on my mark" puts team in ready state. All teams reach ready → player says "Go" → synchronized execution with <100ms delta between first and last agent action.
- 82.7.1.6 Implement movement execution — pathfind to target position using Phase 43 navmesh, with tactical movement (slice the pie around corners, check doorways, cover leapfrog) instead of direct-line movement
- 82.7.1.7 Create fire control execution — "Weapons free" sets ROE to engage all identified hostiles. "Weapons hold" restricts to self-defense. "Suppress [area]" directs sustained fire at area without requiring identified targets.
- 82.7.1.8 Implement posture transitions — "Go quiet" activates stealth behavior tree: suppressed weapons, CQC preference, whispered comms, slower movement. "Go loud" reverses: full auto, shout callouts, sprint.
- 82.7.1.9 Create non-lethal mode — "Non-lethal" restricts squad to tasers, batons, restraints. Hostage rescue scenarios require this.
- 82.7.1.10 Implement command acknowledgment dispatch — when command is received, immediately dispatch acknowledgment to TTS (§82.10): "Copy, moving to position" or "Negative, path blocked"
- 82.7.1.11 Create command failure reporting — if execution fails mid- way (path destroyed, target eliminated, agent down), report failure via TTS and await new orders or fall back to autonomous behavior
- 82.7.1.12 Implement command history — player can say "Again" or "Repeat last" to re-execute the most recent command for a team
- 82.7.1.13 Create tactical command integration tests — scripted breach scenarios verifying correct sequencing, synchronization, and failure handling
82.8 Multi-Agent Formation Control (Rust)#
ORCA-based collision avoidance with Boids-based formation maintenance. Squads maintain tactical formations while navigating through environments, dynamically adjusting spacing for indoor/outdoor transitions.
82.8.1 Formation System#
- 82.8.1.1 Create
maya-formationRust crate with formation templates, ORCA integration, and dynamic spacing - 82.8.1.2 Implement formation template library — wedge (V-shape, balanced), column (single-file, narrow spaces), line abreast (horizontal spread, wide coverage), diamond (360° protection), echelon left/right, staggered column
- 82.8.1.3 Create per-agent formation slot assignment — each team member assigned a relative offset from formation center. Slot assignment considers role (point man at front, medic at center, heavy weapons at rear).
- 82.8.1.4 Implement formation center tracking — formation center follows player (when leading) or designated waypoint (when directed), with smooth acceleration/deceleration matching player speed
- 82.8.1.5 Create ORCA collision avoidance — each squad member runs ORCA velocity obstacle computation against all nearby agents (teammates + civilians + enemies) to prevent collisions while maintaining formation
- 82.8.1.6 Implement adaptive spacing — automatically tighten formation in narrow spaces (hallways, doorways) and expand in open areas, using navmesh width queries to determine available space
- 82.8.1.7 Create bounding overwatch movement — team splits into fire team and maneuver team. Fire team holds and covers. Maneuver team advances to next cover. Roles swap. Alternating advance with mutual support.
- 82.8.1.8 Implement formation transition — smooth interpolation between formation types over 1-2 seconds when player gives formation change command, agents pathfind to new relative positions
- 82.8.1.9 Create formation break and reform — when formation is broken by combat or obstacles, agents attempt to reform at earliest opportunity. If reformation is impossible (separated by walls), maintain local subgroup coherence.
- 82.8.1.10 Implement fire sector auto-assignment — based on formation type and threat direction, each agent automatically assigns a coverage sector (no two agents cover the same sector, full 360° coverage in diamond, forward 180° in wedge)
- 82.8.1.11 Create formation performance benchmark — verify 16-agent formation with ORCA collision avoidance at <0.5ms per frame
- 82.8.1.12 Implement formation visualization — debug overlay showing formation template, agent slots, ORCA velocity obstacles, fire sectors
82.9 Combat AI with Voice Override Priority (Rust)#
Autonomous tactical combat behavior that squad NPCs execute between voice commands. NPCs act like competent soldiers — seeking cover, calling contacts, managing ammo — but always yield to player voice commands.
82.9.1 Autonomous Combat Behavior#
- 82.9.1.1 Implement threat assessment — continuously scan for hostile entities, rate threat level (distance × weapon × stance × visibility), prioritize engagement order (nearest immediate threat first)
- 82.9.1.2 Create cover-seeking behavior — maintain proximity to cover objects (walls, crates, vehicles). When engaged, immediately move to nearest viable cover relative to threat direction. Evaluate cover quality (thickness, height, flanking exposure).
- 82.9.1.3 Implement peek and fire — from cover, lean/peek to acquire target, fire burst, return to cover. Vary peek direction and timing to avoid predictability.
- 82.9.1.4 Create suppressive fire behavior — when teammate is moving, provide covering fire at threat position. Sustain fire for movement duration. Call "Covering!" via TTS.
- 82.9.1.5 Implement ammo management — track magazine state, call "Mag change!" when reloading, seek cover during reload. Call "Running low!" at 25% ammo remaining.
- 82.9.1.6 Create casualty response — when teammate goes down, nearest agent calls "Man down!", moves to downed teammate under covering fire, drags to cover, applies first aid. Player can override with "Leave him, push forward."
- 82.9.1.7 Implement autonomous callouts — "Contact! Three hostiles, ground floor, east side!" using actual spatial relationship to player and compass bearing. Call frequency modulated by personality (taciturn NPCs call less frequently).
- 82.9.1.8 Create room clearing protocol — autonomous CQB: slice the pie, buttonhook or cross-entry at doorways, clear corners, announce "Clear!" or "Hostile down!" per room section
- 82.9.1.9 Implement retreat behavior — when outgunned (threat level exceeds squad capability by 2:1+), propose retreat: "We're outgunned, suggest we pull back!" Player can confirm or override.
- 82.9.1.10 Create noise discipline — in stealth posture, avoid firing unless directly threatened, prefer CQC takedowns, whisper callouts, move slowly
82.9.2 Personality-Driven Combat Variation#
- 82.9.2.1 Implement bravery-driven aggression — high Bravery NPCs push aggressively, volunteer for point position, advance under fire. Low Bravery NPCs stay further back, hesitate before advancing, prefer long-range engagement.
- 82.9.2.2 Create cautiousness-driven thoroughness — high Cautiousness NPCs take extra time to clear corners, double-check rooms, move slowly through doorways. Low Cautiousness NPCs clear faster but may miss threats.
- 82.9.2.3 Implement veteran vs rookie skill progression — rookie NPCs have jerkier movement between cover, slower weapon transitions, wider aim spread. Veterans move fluidly, transition quickly, and have tighter aim. NPCs improve with combat experience over time.
- 82.9.2.4 Create loyalty-driven protective behavior — high Loyalty NPCs prioritize player safety, interpose between player and threats, call warnings about flanking enemies targeting the player. Low Loyalty NPCs prioritize self-preservation.
- 82.9.2.5 Implement discipline-driven command compliance — high Discipline NPCs comply immediately and precisely with voice commands. Low Discipline NPCs may hesitate, comply imperfectly, or question orders ("You sure about that?"). Extremely low Discipline + low Loyalty may refuse suicidal orders.
- 82.9.2.6 Create aggression-driven engagement style — high Aggression NPCs prefer close engagement, push toward enemies, use shotguns/SMGs. Low Aggression NPCs prefer distance, use rifles, maintain defensible positions.
- 82.9.2.7 Implement stress-degraded performance — under sustained combat, NPCs accumulate stress. High Neuroticism NPCs degrade faster: aim worsens, callouts become panicked, movement becomes erratic. High emotional stability NPCs maintain composure under prolonged combat.
- 82.9.2.8 Create personality-driven combat integration tests — verify that personality facets produce visibly different combat behavior across a squad of 4-8 NPCs with diverse personalities
82.9.3 Voice Override Priority System#
- 82.9.3.1 Implement 6-level priority hierarchy — P5: player voice command, P4: self-preservation (under direct fire), P3: team safety (rescue downed), P2: standing order (persistent command), P1: autonomous tactical, P0: idle/formation
- 82.9.3.2 Create priority arbitration — when voice command (P5) arrives, current behavior tree node is suspended (not terminated), voice command node injected, executes to completion, then suspended behavior resumes or is replaced depending on world state
- 82.9.3.3 Implement self-preservation exception — NPC in immediate lethal danger (grenade at feet, melee attack) may delay voice command execution by 0.5-1s to dodge/evade first, then comply. Reports: "Taking cover! Moving to position after!"
- 82.9.3.4 Create compliance personality modulation — high Discipline NPCs comply immediately and precisely. Low Discipline NPCs may hesitate, comply imperfectly, or question orders ("You sure about that?"). Extremely low Discipline + low Loyalty may refuse suicidal orders.
- 82.9.3.5 Implement standing order persistence — "Hold this position" remains active as P2 until countermanded. If NPC is forced to leave position (grenade, fire), returns ASAP and reports "Returning to position."
- 82.9.3.6 Create command queue vs override behavior — sequential commands queue (execute in order). "Override" or urgent tone command replaces queue entirely. System distinguishes by utterance analysis.
- 82.9.3.7 Implement priority system debugging — visualize current priority stack per NPC, active behavior tree node, queued commands
82.10 TTS Feedback & Personality-Matched Voice (TypeScript + Rust)#
NPCs respond vocally to commands and events using text-to-speech synthesis that matches their personality, emotional state, and tactical situation.
82.10.1 Voice Synthesis Pipeline#
- 82.10.1.1 Create
libs/maya/voice-tts/TypeScript library with voice profile management, emotion modulation, and TTS provider abstraction - 82.10.1.2 Implement on-device TTS using Piper TTS (Rust integration) — lightweight neural TTS (<100MB model), <100ms latency for short utterances (callouts, acknowledgments)
- 82.10.1.3 Create voice profile system — each NPC assigned a voice profile specifying: base voice model, pitch range, speaking rate, accent/dialect, vocabulary complexity. Derived from personality facets.
- 82.10.1.4 Implement emotion modulation — current emotional state (Phase 79.24.3) modulates TTS parameters: fear → higher pitch, tremor, faster pace; anger → lower pitch, louder, clipped words; calm → steady pace, neutral pitch; confidence → strong projection
- 82.10.1.5 Create tactical brevity — in combat, NPC voice responses are short and clipped ("Copy!", "Moving!", "Contact left!"). In dialogue mode, full sentences. Context determines verbosity level.
- 82.10.1.6 Implement spatial audio integration — NPC voice responses spatialized in 3D using Phase 79.38 audio propagation: NPC at distance sounds distant, NPC behind wall sounds muffled, NPC on radio sounds filtered
- 82.10.1.7 Create voice priority queue — multiple NPCs speaking simultaneously are serialized with priority (command acknowledgment > contact callout > status update > ambient chatter) to prevent audio overload
- 82.10.1.8 Implement lip sync integration — TTS audio output drives Phase 79.17 viseme generation for lip-synchronized facial animation
- 82.10.1.9 Create voice variety system — ensure squad members have distinct voices so player can identify who is speaking without seeing them
- 82.10.1.10 Implement TTS caching — frequently used callouts ("Copy", "Moving", "Contact!", "Clear!") pre-generated and cached at session start for instant playback without synthesis latency
- 82.10.1.11 Create radio filter effect — when NPC is out of visual range, apply radio comm filter (bandwidth limiting, static noise) to TTS output to simulate radio communication
- 82.10.1.12 Implement TTS quality benchmark — verify <150ms latency for callouts, <500ms for full sentences, natural prosody, personality distinguishability between NPC voices
82.11 Wake Word & Voice Activation Modes (Rust)#
Three voice input modes for different gameplay situations, ensuring voice control feels natural and never interferes with gameplay when not intended.
82.11.1 Activation System#
- 82.11.1.1 Implement push-to-talk mode — designated key/button activates microphone. Default mode. Zero false activation risk. Released key = end of utterance.
- 82.11.1.2 Create wake word mode using Picovoice Porcupine — custom wake words: "Hey [NPC name]" for conversation, "Squad" or "Team" for tactical commands. <10KB per keyword, <30ms detection, fully on-device.
- 82.11.1.3 Implement proximity auto-listen — when player is within 2m of an NPC in dialogue mode, microphone stays active (with clear on-screen indicator). Deactivates when player moves away or conversation ends.
- 82.11.1.4 Create mode switching — player can switch between modes via settings or in-game command. Modes can be assigned per context (PTT for combat, wake word for exploration, auto-listen for dialogue).
- 82.11.1.5 Implement false activation prevention — wake word + VAD confirmation required (must detect actual speech after wake word, not just ambient noise matching)
- 82.11.1.6 Create voice input indicator UI — persistent but unobtrusive HUD element showing current mode (PTT/wake/auto), microphone state (listening/processing/recognized), and confidence level
- 82.11.1.7 Implement voice input routing — based on game context, route transcribed speech to appropriate handler: tactical command → §82.2/82.3, NPC dialogue → §82.5, negotiation → §82.6, general → context analysis
- 82.11.1.8 Create custom wake word training — players can assign custom names to squad members, Porcupine trains custom keyword model on first use
- 82.11.1.9 Implement voice input accessibility — configurable sensitivity for different speech volumes, support for speech impediments via adjusted recognition thresholds, visual command input as always-available fallback
- 82.11.1.10 Create voice activation integration tests — test all three modes in gameplay scenarios: PTT during combat, wake word during exploration, auto-listen during NPC dialogue
Phase 82 Summary — Voice-as-First-Class Gameplay Input#
New Systems:
| System Area | Section | Tasks |
|---|---|---|
| Core ASR Engine | 82.1.1 | 12 |
| Intent Classifier | 82.2.1 | 12 |
| SLM Command Planner | 82.3.1 | 10 |
| Spatial Resolver | 82.4.1 | 12 |
| Voice Dialogue System | 82.5.1 | 15 |
| Negotiation Engine | 82.6.1 | 12 |
| Command Execution | 82.7.1 | 13 |
| Formation System | 82.8.1 | 12 |
| Autonomous Combat Behavior | 82.9.1 | 10 |
| Personality Combat Variation | 82.9.2 | 8 |
| Voice Override Priority | 82.9.3 | 7 |
| Voice Synthesis Pipeline | 82.10.1 | 12 |
| Activation System | 82.11.1 | 10 |
| TOTAL Phase 82 | 145 |
Voice Capabilities After Phase 82#
| Capability | Before Phase 82 | After Phase 82 |
|---|---|---|
| Player voice input | Modding companion only (Phase 78) | Full gameplay voice control |
| NPC conversation | Text/scripted dialogue | Natural voice conversation with any NPC |
| Squad commands | Menu/button-based | Natural language tactical voice commands |
| Negotiation | Dialogue tree options | Free-form voice persuasion/intimidation/deception |
| Command latency | N/A | <650ms end-to-end (matching F.A.C.U.L.) |
| Spatial references | N/A | Gaze + CLIP + entity DB resolution |
| Squad formations | Manual positioning | Voice-commanded tactical formations |
| Combat AI | Basic behavior trees | Full autonomous tactical behavior with voice override |
| NPC voice responses | Pre-recorded barks | Personality-matched TTS with emotion |
| Activation modes | N/A | PTT, wake word, proximity auto-listen |
Combined with Phase 81's cognitive core and Phase 79's NPC life simulation, Phase 82 makes every NPC in the Maya world someone the player can talk to — and every squad member someone the player can command with their voice in real-time tactical combat.