Historical task source. The Eve task board holds current task state.
Phase 81: NPC Cognitive Core & On-Device SLM — Tiered NPC Intelligence#
Purpose: Build the cognitive infrastructure that makes every NPC in the Maya engine a coherent thinking agent — from the 10,000+ background NPCs running lightweight personality neural networks, through mid-range NPCs running on-device 2B SLMs, to close-range NPCs with full 4-8B SLM cognition with episodic memory and reflection. This phase unifies all existing NPC subsystems (Phase 43 behavior trees, Phase 79.24 needs/personality/emotions/memory, Phase 79.25 relationships, Phase 40.10 LLM integration) into a single perception→remember→think→act cognitive loop with seamless LOD transitions between cognition tiers.
Reference Proposal:
docs/proposals/LIVING_WORLD_NPC_SOTA_PROPOSAL.md
Scope: Rust crates in libs/maya/engine-core/crates/ for the performance-
critical cognitive loop, personality neural network, attention system, and
cognitive LOD manager. TypeScript libraries in libs/maya/ for SLM prompt
orchestration, memory/RAG services, cloud LLM fallback, and teacher-student
distillation pipeline. Integration with llama.cpp (via FFI) and ONNX Runtime for
on-device model inference.
Technology:
- Rust crates (in
libs/maya/engine-core/crates/):maya-cognition(core loop),maya-personality-nn(lightweight neural network),maya-attention(salience scoring),maya-cognitive-lod(tier management),maya-slm-runtime(llama.cpp FFI wrapper),maya-vram-budget(GPU memory allocation) - TypeScript libraries (in
libs/maya/):npc-cognition-orchestrator(prompt management, RAG, reflection),npc-distillation(teacher-student pipeline),npc-cloud-fallback(hybrid cloud reasoning) - External dependencies: llama.cpp (C++ via FFI), ONNX Runtime (inference), E5-Large-Unsupervised (embeddings)
- Integration targets: Phase 43 (behavior trees, navmesh), Phase 79.24 (NPC life simulation), Phase 79.25 (relationships), Phase 79.31 (NPC-NPC interaction), Phase 79.32 (simulation LOD), Phase 40.10 (maya/souls)
81.1 Personality Neural Network — Tier 2 Lightweight Brain (Rust)#
Lightweight neural network executing on CPU for mid-range NPCs (15-100m). Each NPC's personality facets from Phase 79.24.2 are compressed to a 12-dimensional input vector that drives a small feedforward network producing 8 action-category scores. Runs at <0.01ms per NPC, enabling 5,000+ simultaneous mid-range NPCs on 4 CPU cores.
81.1.1 Neural Network Architecture#
- 81.1.1.1 Create
maya-personality-nnRust crate with Cargo.toml, module structure, benches directory, and integration tests directory - 81.1.1.2 Implement feedforward network structure — 12 input neurons, two hidden layers of 32 neurons each with ReLU activation, 8 output neurons with softmax producing action-category probabilities
- 81.1.1.3 Create SIMD-optimized matrix multiplication for forward pass
using
std::simdorpacked_simd— target <0.005ms per inference on single core - 81.1.1.4 Implement input encoding — compress Phase 79.24.2 personality facets (30+ dimensions) to 12D via principal component analysis: 5 Big Five scores, 3 most-deficient need values, weather/time/danger context, social proximity density
- 81.1.1.5 Create output action categories — continue_activity, seek_need, initiate_social, avoid_threat, pursue_goal, idle_wander, emotional_reaction, flee — each with continuous probability score
- 81.1.1.6 Implement weighted random selection from top-3 scoring actions with temperature parameter (prevents robotic always-optimal behavior, matches Phase 79.24.1.6 design)
- 81.1.1.7 Create batch inference engine — process up to 256 NPCs in a single vectorized forward pass using SIMD-wide matrix operations
- 81.1.1.8 Implement weight serialization — load/save trained weights in compact binary format (<10KB per personality archetype)
- 81.1.1.9 Create personality archetype weight library — 50+ pre-trained weight sets matching Phase 79.24.2.6 archetype templates (warrior, scholar, merchant, healer, trickster, noble, etc.)
- 81.1.1.10 Implement per-NPC weight interpolation — blend between two nearest archetype weight sets based on personality distance, enabling unique behavior per NPC without unique training
- 81.1.1.11 Create context injection mechanism — environmental context (rain, night, combat nearby, festival) modulates input vector to produce weather-responsive and event-responsive behavior changes
- 81.1.1.12 Implement decision rate limiter — Tier 2 NPCs make new decisions every 1-4 seconds (not every frame), with current action persisting between decisions
- 81.1.1.13 Create decision logging for distillation — record all (input, output, selected_action) tuples for teacher-student training pipeline (§81.4)
- 81.1.1.14 Implement performance benchmark — verify <0.01ms per NPC inference, 5,000+ batch at <4ms total, memory <10KB per NPC
- 81.1.1.15 Create personality NN debugging overlay — visualize input encoding, hidden activations, output probabilities per selected NPC
81.2 On-Device SLM Inference Engine (Rust)#
Wraps llama.cpp via FFI to run Gemma 4 E2B/E4B or Nemotron-Mini-4B models inside the game process. Provides batched inference, KV-cache management, and frame-synchronized scheduling so SLM computation interleaves with rendering without frame drops.
81.2.1 llama.cpp Integration#
- 81.2.1.1 Create
maya-slm-runtimeRust crate with safe FFI wrapper around llama.cpp C API (llama_model_load, llama_context_new, llama_decode, llama_sampling) - 81.2.1.2 Implement model loading with quantization support — Q4_K_M, Q5_K_M, Q8_0 for different quality/performance tradeoffs
- 81.2.1.3 Create model registry — register available SLM models with metadata (parameter count, VRAM requirement, token generation rate, personality specialization)
- 81.2.1.4 Implement KV-cache pool management — pre-allocate KV-cache slots for N concurrent NPC contexts, reclaim when NPC demotes to lower tier
- 81.2.1.5 Create context window management — per-NPC context containing personality system prompt + retrieved memories + current percepts + recent dialogue, trimmed to fit model context length
- 81.2.1.6 Implement token-by-token generation with frame budget integration — generate K tokens per frame (configurable), yielding control back to renderer between token groups
- 81.2.1.7 Create multi-NPC batch inference — group pending NPC queries and process in a single forward pass where possible (continuous batching)
- 81.2.1.8 Implement prompt template system — parameterized templates for different NPC query types: decision_making, dialogue_response, planning, reflection, reaction
- 81.2.1.9 Create structured output parsing — SLM responses parsed into typed action structs (ActionDecision, DialogueResponse, PlanUpdate, ReflectionInsight) with validation
- 81.2.1.10 Implement generation constraints — constrained decoding limiting SLM output to valid game actions, preventing hallucinated actions that don't exist in the game
- 81.2.1.11 Create inference cancellation — when an NPC demotes tier or the query becomes stale (NPC died, player left), cancel in-progress generation immediately and reclaim resources
- 81.2.1.12 Implement model hot-swap — switch between models (e.g., 2B to 4B) without restarting the engine, for runtime quality scaling
- 81.2.1.13 Create SLM inference profiling — per-query timing (time to first token, tokens per second, total latency), integrated with Tracy
- 81.2.1.14 Implement fallback to CPU inference — when GPU VRAM is exhausted, transparently fall back to CPU inference at reduced throughput
- 81.2.1.15 Create SLM runtime integration tests — verify correct responses for known personality + context combinations
81.2.2 Model Configuration & Tiering#
- 81.2.2.1 Implement Tier 1 model configuration — 2B parameter model (Gemma 4 E2B Q4_K_M, ~1.2GB VRAM) for quick decisions, barks, and reactions within 15m range
- 81.2.2.2 Create Tier 0 model configuration — 4-8B parameter model (Gemma 4 E4B Q4_K_M, ~2.5GB VRAM or Nemotron-Mini-4B, ~2GB VRAM) for full dialogue, complex reasoning, and planning for interacting NPCs
- 81.2.2.3 Implement model selection heuristic — NPC importance score (distance, narrative role, player attention, relationship) determines which model tier an NPC receives
- 81.2.2.4 Create LoRA adapter support — load per-NPC or per-archetype LoRA adapters that specialize the base model's personality without full model duplication (rank 8-16, ~50MB per adapter)
- 81.2.2.5 Implement companion character dedicated context — key story NPCs (party members, recurring allies) receive a permanently loaded KV-cache that persists across scenes for conversational continuity
- 81.2.2.6 Create multi-model simultaneous loading — Tier 1 (2B) and Tier 0 (4B) models loaded simultaneously, with separate inference queues
- 81.2.2.7 Implement model warmup on level load — pre-fill KV-caches for narrative-critical NPCs before gameplay begins
- 81.2.2.8 Create quality preset system — "Performance" (Tier 1 only, 2B), "Balanced" (Tier 0 + Tier 1, 4B + 2B), "Ultra" (Tier 0 + Tier 1, 8B + 4B) with automatic selection based on hardware detection
81.3 VRAM Budget Manager & Compute Scheduling (Rust)#
Manages GPU memory allocation across SLM inference, rendering, and other GPU workloads. Implements Compute-in-Graphics (CIG) scheduling inspired by NVIDIA ACE to run AI inference between render passes without frame drops.
81.3.1 VRAM Allocation#
- 81.3.1.1 Create
maya-vram-budgetRust crate with GPU memory tracking, allocation policies, and eviction strategies - 81.3.1.2 Implement GPU memory census — detect total VRAM, query current usage by renderer, and compute available SLM budget at startup and on quality preset change
- 81.3.1.3 Create SLM VRAM reservation system — reserve fixed VRAM blocks for model weights (immutable) and KV-cache pool (managed) at load time
- 81.3.1.4 Implement KV-cache slot allocator — fixed-size slots (256-token context per slot), pool of N slots, LRU eviction when pool is exhausted
- 81.3.1.5 Create VRAM pressure monitoring — track allocation rate, detect approaching OOM, and proactively demote NPCs from Tier 0→1→2 to shed load
- 81.3.1.6 Implement dynamic quality scaling — when VRAM pressure exceeds threshold, reduce SLM quantization level (Q8→Q5→Q4), reduce max concurrent Tier 0 NPCs, or switch to smaller model
- 81.3.1.7 Create VRAM reporting dashboard — real-time breakdown of VRAM usage by category (renderer, SLM weights, KV-caches, textures, audio)
- 81.3.1.8 Implement memory defragmentation — compact KV-cache pool when fragmentation exceeds threshold, moving active caches to contiguous memory
81.3.2 Compute-in-Graphics Scheduling#
- 81.3.2.1 Implement CIG scheduler — identify GPU idle bubbles between render passes (shadow map → GI → post-process transitions) and schedule SLM inference work items into these gaps
- 81.3.2.2 Create frame budget partitioning — allocate configurable fraction of frame time to AI compute (default 25% = 4.15ms @ 60fps)
- 81.3.2.3 Implement preemption-aware scheduling — if SLM inference exceeds its time slice, pause and resume next frame rather than causing frame drop
- 81.3.2.4 Create async compute queue integration — submit SLM inference as async compute work that runs in parallel with graphics pipeline stages that don't contend for the same GPU resources
- 81.3.2.5 Implement priority queue for SLM queries — interacting NPCs (Tier 0) get priority over ambient NPCs (Tier 1), narrative-critical NPCs get priority over all
- 81.3.2.6 Create CPU↔GPU synchronization — when SLM inference completes, signal the cognitive loop on CPU to process the result and update NPC state
- 81.3.2.7 Implement multi-frame inference — long SLM queries (planning, reflection) span multiple frames, with intermediate tokens cached and final result delivered when complete
- 81.3.2.8 Create CIG profiling — measure actual GPU utilization overlap between rendering and AI inference, identify scheduling inefficiencies
81.4 Teacher-Student Cognition Distillation (TypeScript + Rust)#
The pipeline that makes the whole tiered system coherent: full SLM decisions (Tier 0/1) are logged as training data, and periodically the Tier 2 personality neural networks are fine-tuned on these decisions. This means Tier 2 NPCs make decisions that are statistically consistent with what the SLM would have decided — without SLM cost.
81.4.1 Training Data Pipeline#
- 81.4.1.1 Create
libs/maya/npc-distillation/TypeScript library with project.json, package.json, tsconfig.json, and vitest config - 81.4.1.2 Implement SLM decision logger — capture every Tier 0/1 SLM decision as a (personality_vector, context_encoding, action_selected) tuple with timestamp and NPC archetype tag
- 81.4.1.3 Create decision buffer — ring buffer storing last 100,000 decisions in memory, flushed to disk periodically for offline training
- 81.4.1.4 Implement context encoding compressor — reduce full SLM context (personality + needs + percepts + memories) to the 12D input vector that the personality NN consumes, ensuring dimensional alignment
- 81.4.1.5 Create archetype-stratified sampling — ensure training data is balanced across personality archetypes, over-sampling rare archetypes to prevent majority-class bias
- 81.4.1.6 Implement training data quality filter — discard decisions where SLM confidence was low or output was malformed, retaining only high-quality training signal
- 81.4.1.7 Create training data versioning — tag data with model version, game version, and distillation epoch to track data provenance
81.4.2 Distillation Training Loop#
- 81.4.2.1 Implement offline distillation trainer — given a batch of (input, action) pairs, update personality NN weights via backpropagation with cross-entropy loss between NN output and SLM-selected action
- 81.4.2.2 Create per-archetype fine-tuning — train separate weight sets for each personality archetype from SLM decisions made by NPCs of that archetype
- 81.4.2.3 Implement incremental training — update weights without full retraining, processing new decision batches as they accumulate
- 81.4.2.4 Create validation split — hold out 20% of decisions for validation, measure action prediction accuracy, track improvement over distillation epochs
- 81.4.2.5 Implement weight deployment — push newly trained NN weights to the game runtime, hot-swappable without restart
- 81.4.2.6 Create distillation metrics dashboard — track per-archetype accuracy, loss curves, decision distribution alignment between SLM and NN
- 81.4.2.7 Implement automated distillation schedule — trigger retraining when sufficient new data has accumulated (configurable threshold, default 10,000 new decisions per archetype)
- 81.4.2.8 Create A/B testing framework — run a fraction of Tier 2 NPCs on old weights and new weights simultaneously, compare behavioral coherence metrics
81.5 Unified Cognitive Loop (Rust)#
The core perception→remember→think→act cycle that runs for every Tier 0 and Tier 1 NPC every tick. Integrates all existing NPC subsystems into a single coherent pipeline.
81.5.1 Perception Stage#
- 81.5.1.1 Create
maya-cognitionRust crate with cognitive loop state machine, ECS component definitions, and system scheduling - 81.5.1.2 Implement visual perception — query spatial index for entities within NPC's field of view (configurable cone, default 120° horizontal, 60° vertical) with line-of-sight occlusion checks
- 81.5.1.3 Create auditory perception — query audio events within hearing range (configurable, default 30m quiet / 100m loud), attenuated by distance and occlusion (walls, doors)
- 81.5.1.4 Implement game state perception — receive relevant game events (time changes, weather changes, nearby quest triggers, economy events) filtered by NPC's location and relevance
- 81.5.1.5 Create percept struct — unified representation: (entity_or_event, position, type, salience_score, timestamp) stored in NPC's working memory for current tick
- 81.5.1.6 Implement percept deduplication — if the same entity was perceived last tick and hasn't changed, don't re-process (only process new percepts and changed states)
- 81.5.1.7 Create perception LOD — Tier 0 NPCs perceive 360° at full resolution, Tier 1 NPCs perceive forward cone only, Tier 2 NPCs perceive only high-salience events
- 81.5.1.8 Implement perception budget — maximum 20 percepts per tick per NPC, sorted by salience, excess discarded
81.5.2 Memory Stage#
- 81.5.2.1 Implement memory formation — new percepts above importance threshold stored as episodic memories in NPC's memory stream (integrates with Phase 79.24.4)
- 81.5.2.2 Create memory retrieval — given current percepts and active
goals, retrieve top-K relevant memories using scoring function:
score = α×recency + β×importance + γ×relevance(Stanford Generative Agents formula) - 81.5.2.3 Implement embedding-based relevance scoring — compute cosine similarity between current situation embedding and stored memory embeddings (on-device E5-Large-Unsupervised or SLM-generated embeddings)
- 81.5.2.4 Create recency decay — exponential decay (configurable rate, default 0.995 per game-hour) applied to memory recency score since last access
- 81.5.2.5 Implement importance auto-rating — routine events (walking, eating) rated 1-3, notable events (meeting player, finding item) rated 4-6, life-changing events (death of friend, marriage, betrayal) rated 7-10
- 81.5.2.6 Create working memory assembly — combine current percepts + retrieved episodic memories + active emotional state + active plan into a working memory buffer that feeds the THINK stage
- 81.5.2.7 Implement memory capacity enforcement — prune oldest/lowest- importance memories when stream exceeds capacity (Phase 79.24.4.5: 200-500 memories)
- 81.5.2.8 Create reflection trigger check — if cumulative importance of new memories since last reflection exceeds threshold (default 150), schedule reflection task for SLM processing (§81.5.3)
81.5.3 Thinking Stage#
- 81.5.3.1 Implement plan evaluation — check if current plan (daily schedule from Phase 79.24) still makes sense given current percepts and emotional state. If plan is invalidated (e.g., workplace on fire), flag for replanning.
- 81.5.3.2 Create decision request formation — assemble prompt for SLM containing: personality summary, current emotional state, active needs, retrieved memories, current percepts, active plan, and question ("What should I do next?" or "How should I respond to this?")
- 81.5.3.3 Implement Tier 0 thinking — submit decision request to 4-8B SLM (§81.2), await structured response containing selected action, reason, and emotional reaction
- 81.5.3.4 Create Tier 1 thinking — submit decision request to 2B SLM (§81.2), with simplified prompt (shorter context, fewer retrieved memories), faster but less nuanced
- 81.5.3.5 Implement Tier 2 thinking — bypass SLM entirely, run personality NN (§81.1) on encoded input, map output action category to specific behavior tree node
- 81.5.3.6 Create reflection execution — when triggered (§81.5.2.8), SLM generates high-level questions about recent experiences, retrieves relevant memories, synthesizes insights stored back as high-importance memories
- 81.5.3.7 Implement replanning — when current plan is invalidated, SLM generates a new plan for the remainder of the day incorporating current circumstances, emotional state, and long-term goals
- 81.5.3.8 Create emotional update — THINK stage output includes emotional delta (e.g., fear +15, anger +5), applied to Phase 79.24.3 emotional state
- 81.5.3.9 Implement social reasoning — when another NPC or the player is perceived, evaluate relationship (Phase 79.25), decide whether to initiate interaction, and select interaction type
- 81.5.3.10 Create thinking budget enforcement — if SLM queue is full, Tier 0/1 NPCs fall back to Tier 2 personality NN decision for this tick, with SLM query deferred to next available slot
81.5.4 Action Stage#
- 81.5.4.1 Implement action dispatch — map THINK stage output (action type + parameters) to Phase 43 behavior tree node activation
- 81.5.4.2 Create voice command override injection — if a player voice command targets this NPC (from Phase 82), inject it as Priority 5 node that overrides current THINK output
- 81.5.4.3 Implement emotional modulation of action — emotional state affects animation style (Phase 79.17 facial expression, body language), movement speed, voice tone
- 81.5.4.4 Create social context adjustment — if NPC is in a group, adjust action parameters (speak quieter in library, louder in tavern, follow group movement)
- 81.5.4.5 Implement action logging — record executed action as memory for future retrieval ("I decided to go to the tavern because I was lonely")
- 81.5.4.6 Create action failure handling — if behavior tree execution fails (path blocked, target gone), log failure and trigger re-evaluation next tick
- 81.5.4.7 Implement action animation integration — selected action maps to animation state in Phase 79.2 motion matching system
- 81.5.4.8 Create cognitive loop profiling — per-stage timing (perceive, remember, think, act) per NPC, integrated with Tracy
81.6 Attention System & Salience Scoring (Rust)#
Determines what each NPC notices and pays attention to. Without this, NPCs would either process everything (too expensive) or nothing (unresponsive). The attention system makes NPCs feel alive by having them notice what matters.
81.6.1 Salience Model#
- 81.6.1.1 Create
maya-attentionRust crate with salience scoring pipeline and attention budget manager - 81.6.1.2 Implement base salience scoring — each percept receives a
score from:
salience = loudness×0.3 + motion×0.25 + size×0.1 + novelty×0.2 + emotional_relevance×0.15(weights configurable) - 81.6.1.3 Create novelty detection — entities seen for the first time or entities that have changed state since last observation receive novelty bonus
- 81.6.1.4 Implement emotional relevance — entities related to NPC's fears (high fear → enemy entity scores higher), loves (family member scores higher), goals (quest objective scores higher)
- 81.6.1.5 Create change detection — track entity states frame-to-frame, flag any changes (moved, health changed, started animation, drew weapon) for high-priority attention
- 81.6.1.6 Implement threat detection fast-path — entities flagged as hostile, drawing weapons, or approaching aggressively bypass normal salience scoring and receive maximum attention priority
- 81.6.1.7 Create attention persistence — once an NPC attends to something, it maintains attention for a minimum duration (1-5 seconds) before switching, preventing jittery attention
- 81.6.1.8 Implement attention memory — what the NPC was attending to last tick influences this tick (continuity of focus), with gradual disengagement if stimulus fades
- 81.6.1.9 Create social attention — NPCs notice when other NPCs are attending to something and orient toward the same stimulus (shared attention, prerequisite for crowd reactions)
- 81.6.1.10 Implement personality-modulated attention — high Openness → attends to novel stimuli more readily; high Neuroticism → attends to threats more readily; high Extraversion → attends to social stimuli more
- 81.6.1.11 Create attention visualization — debug overlay showing each NPC's attention focus as a colored line/cone, with salience scores for all percepts
- 81.6.1.12 Implement attention budget — configurable max percepts per tick per tier (Tier 0: 15, Tier 1: 8, Tier 2: 3), excess percepts discarded by lowest salience
81.7 Cloud LLM Fallback & Hybrid Reasoning (TypeScript)#
For complex multi-NPC reasoning (political schemes, group planning, trade negotiations, narrative-critical decisions) that exceeds on-device SLM capacity, a cloud LLM (Claude, Gemini, or similar) is queried asynchronously.
81.7.1 Cloud Fallback Pipeline#
- 81.7.1.1 Create
libs/maya/npc-cloud-fallback/TypeScript library with project.json, package.json, tsconfig.json, and vitest config - 81.7.1.2 Implement cloud query manager — queue of pending cloud LLM requests, prioritized by narrative importance, with configurable max concurrent requests and rate limiting
- 81.7.1.3 Create multi-NPC reasoning requests — when multiple NPCs need to coordinate (faction war council, merchant guild meeting, family discussion), batch them into a single cloud query with all participants' personalities, memories, and goals
- 81.7.1.4 Implement latency-tolerant design — cloud queries are asynchronous (1-5 second response time), NPC continues with Tier 1/2 behavior while awaiting result, then integrates cloud response when it arrives
- 81.7.1.5 Create result caching — cache cloud LLM responses for similar query patterns, reducing redundant API calls for common social scenarios
- 81.7.1.6 Implement result distillation — cloud LLM decisions are logged into the teacher-student pipeline (§81.4) so on-device models learn from cloud reasoning over time
- 81.7.1.7 Create offline mode — when no internet connection is available, gracefully degrade to on-device SLM only, with reduced cognitive depth for complex multi-NPC scenarios
- 81.7.1.8 Implement cloud safety layer — filter cloud LLM responses for content safety, game-world consistency (no hallucinated items/locations), and narrative coherence before applying to game state
- 81.7.1.9 Create cloud usage analytics — track per-session API usage, cost, latency distribution, cache hit rate, and quality improvement over on-device decisions
- 81.7.1.10 Implement provider abstraction — support Claude, Gemini, GPT, and self-hosted models (vLLM) behind a unified interface, with automatic provider selection based on latency, cost, and availability
81.8 Cognitive LOD Manager (Rust)#
The master system that decides which cognition tier each NPC runs at, manages tier transitions with state preservation, generates plausible alibis for NPCs promoting from low-detail to high-detail tiers, and enforces overall CPU and GPU budgets across all cognitive systems.
81.8.1 Tier Management#
- 81.8.1.1 Create
maya-cognitive-lodRust crate integrating with Phase 79.32 simulation LOD as a cognitive overlay - 81.8.1.2 Implement tier assignment — per-frame scoring of all NPCs:
tier_score = distance_weight×(1/distance) + attention_weight×player_gaze + narrative_weight×quest_importance + interaction_weight×in_dialogue, top N NPCs assigned to available Tier 0/1 slots - 81.8.1.3 Create hysteresis bands — NPC promotes to Tier 0 at 10m but demotes at 18m (not 15m), preventing flickering at boundary
- 81.8.1.4 Implement promotion with context generation — when NPC promotes from Tier 2→1 or 1→0, generate SLM context by assembling personality + schedule-based alibi + recent memories + current needs into initial prompt
- 81.8.1.5 Create demotion with state snapshot — when NPC demotes from Tier 0→1 or 1→2, save current SLM context summary, emotional state, active plan, and last decision to enable coherent resumption if promoted again
- 81.8.1.6 Implement alibi generation — when a Tier 3/4 NPC (schedule- driven or statistical) promotes to Tier 2+, generate a plausible "what I've been doing" narrative from schedule template + occupation + time of day, so the NPC has coherent recent history
- 81.8.1.7 Create narrative lock — quest-critical NPCs are locked to Tier 0 regardless of distance, ensuring they always have full cognition when the player can potentially interact with them
- 81.8.1.8 Implement companion dedication — player's party member NPCs receive permanently dedicated Tier 0 slots with persistent KV-caches that survive scene transitions
- 81.8.1.9 Create tier distribution reporting — real-time display of how many NPCs are at each tier, total CPU/GPU budget utilization, and slot availability
- 81.8.1.10 Implement budget enforcement — if total cognitive compute exceeds frame budget, force-demote lowest-priority Tier 0→1→2 NPCs until budget is met
- 81.8.1.11 Create tier transition smoothing — cognitive output doesn't change abruptly on tier transition; blend old and new tier outputs for 2-3 ticks during transition
- 81.8.1.12 Implement stress testing — spawn 10,000+ NPCs across all tiers, verify budget compliance under worst-case scenarios (player teleporting, combat with many visible NPCs)
- 81.8.1.13 Create cognitive LOD integration with Phase 79.32 simulation LOD — cognitive tiers align with simulation tiers but can differ (a narratively important NPC at 50m may be Tier 1 cognitive but Tier 1 simulation)
- 81.8.1.14 Implement save/load for cognitive state — all NPC tier assignments, KV-cache contents, and pending SLM queries serialize correctly on game save and restore on load
- 81.8.1.15 Create cognitive LOD documentation and tuning guide for designers — explain tier system, budget knobs, and quality tradeoffs
Phase 81 Summary — NPC Cognitive Core & On-Device SLM#
New Systems:
| System Area | Section | Tasks |
|---|---|---|
| Personality Neural Network | 81.1.1 | 15 |
| llama.cpp Integration | 81.2.1 | 15 |
| Model Configuration & Tiering | 81.2.2 | 8 |
| VRAM Allocation | 81.3.1 | 8 |
| CIG Scheduling | 81.3.2 | 8 |
| Training Data Pipeline | 81.4.1 | 7 |
| Distillation Training Loop | 81.4.2 | 8 |
| Perception Stage | 81.5.1 | 8 |
| Memory Stage | 81.5.2 | 8 |
| Thinking Stage | 81.5.3 | 10 |
| Action Stage | 81.5.4 | 8 |
| Attention & Salience | 81.6.1 | 12 |
| Cloud Fallback | 81.7.1 | 10 |
| Cognitive LOD Manager | 81.8.1 | 15 |
| TOTAL Phase 81 | 140 |
Cognitive Architecture After Phase 81#
| Capability | Before Phase 81 | After Phase 81 |
|---|---|---|
| NPC decision making | Cloud LLM or behavior tree | 5-tier cognitive LOD from statistical to full SLM |
| On-device inference | None | Gemma/Nemotron 2-8B running in-process |
| NPC count with cognition | ~50 (cloud LLM budget) | 10,000+ (Tier 2 NN) + 10-20 (Tier 0/1 SLM) |
| Decision coherence across tiers | N/A | Teacher-student distillation ensures Tier 2 matches SLM |
| Memory-informed decisions | Separate memory system | RAG-augmented SLM decisions with episodic retrieval |
| Attention modeling | None | Personality-modulated salience scoring |
| VRAM management | None | Budget manager with CIG scheduling |
| Cognitive persistence | None | KV-cache preservation across scene transitions |
| Cloud reasoning | Primary path | Fallback for complex multi-NPC scenarios |
Combined with Phase 79's NPC life simulation and Phase 43's behavior tree execution, Phase 81 gives every NPC in the Maya world a brain — tiered from lightweight personality-driven reflexes to full LLM-powered cognition with memory, reflection, and planning.