# Nous Domain — Feature Reference Nous is the sovereign AI inference backbone of the Oshun ecosystem. It enables any application in the monorepo to run large language models locally on any available hardware, generate and search vector embeddings, build autonomous agents with planning and memory, process documents into searchable vector indexes, engineer prompts with full template systems, and perform local computer vision, audio, and generative-media inference — all without depending on external cloud AI services. Nous runs on the same server that hosts the application, keeping data private, eliminating per-token API costs, and enabling operation in air-gapped or restricted-network environments. Nous is a **library-only domain** — no apps, no standalone services. It ships nineteen implemented TypeScript workspace packages under `libs/nous/` (plus one config-only scaffold, `@nous/platform`), all consumed directly as source. **Library prefix:** `@nous/*` | Library | Package | Source modules | Primary Focus | | ----------------- | ---------------- | -------------- | ------------------------------------------------------------------------------- | | **nous-core** | `@nous/core` | 84 | Inference runtime, hardware acceleration, model management, embeddings, search | | **nous-llm** | `@nous/llm` | 78 | LLM completion APIs, agents, prompt engineering, document processing, RAG | | **nous-vision** | `@nous/vision` | 103 | Image and video computer vision, image/video generation, diffusion, OCR | | **nous-audio** | `@nous/audio` | 81 | Speech recognition, speech synthesis, music generation, audio analysis | | **nous-training** | `@nous/training` | 137 | Distributed training, fine-tuning, preference alignment, diffusion training | | **nous-safety** | `@nous/safety` | 64 | Safety classifiers, output enforcement, interpretability, V2 Model Card hosting | Eight further packages cover controllable generative media (`@nous/image-control`, `@nous/video-control`, `@nous/video-removal`, `@nous/video-to-audio`, `@nous/advanced-speech`, `@nous/generative-relighting`, `@nous/portrait-animation`, `@nous/joint-av-generation`), and four provide cooperative-intelligence primitives for the Concordia domain (`@nous/preference-inference`, `@nous/agreement-search`, `@nous/cooperative-bargaining`, `@nous/concordia-sealed-memory`). A full package inventory with module counts is in `DOMAINS/nous/specifications.md` §1. --- ## 1. Hardware-Accelerated Inference Nous abstracts the ONNX Runtime (a cross-platform machine learning inference engine) behind a unified provider interface. Applications can run models on any available hardware — from a laptop CPU to a multi-GPU server — without any code changes. The runtime tries providers in priority order and falls back automatically. ### 1.1 Execution Providers | Capability | Provider | What It Enables | | --------------------------- | -------- | --------------------------------------------------------------------- | | **CPU inference** | `cpu` | Universal fallback; runs on any machine, no GPU drivers needed | | **NVIDIA GPU inference** | `cuda` | High-throughput inference on NVIDIA GeForce, RTX, A100, H100 | | **AMD GPU inference** | `rocm` | Inference on AMD Radeon RX and Instinct MI-series GPUs | | **Apple GPU inference** | `metal` | Metal-accelerated inference on Apple Silicon (M1/M2/M3) and Intel Mac | | **Browser GPU inference** | `webgpu` | In-browser GPU inference via the WebGPU API (Chrome, Firefox) | | **Cross-platform GPU** | `vulkan` | Vulkan-based GPU inference on Linux, Windows, and Android | | **Windows GPU inference** | `dml` | DirectML inference on any DirectX 12-capable Windows GPU | | **Neural Processing Units** | `npu` | Inference on dedicated NPU and TPU hardware accelerators | Each provider accepts device selection (e.g., GPU device index 0 vs. 1), provider-specific tuning parameters, and configures graceful fallback to the next available provider. ### 1.2 Provider Routing and Fallback - **Ranked provider ordering**: Specify an ordered list of preferred providers (e.g., `["cuda", "cpu"]`). The runtime tries each in sequence, using the first that initializes successfully. - **Automatic fallback**: Provider failures, driver unavailability, and model compatibility issues are handled transparently — the application never needs to handle hardware-specific errors. - **Configuration validation**: Invalid provider configurations (e.g., negative GPU device IDs, unsupported quantization for a provider) are rejected at configuration time with a clear error, before any inference attempt. - **Dynamic runtime loading**: ONNX Runtime is loaded lazily at first use. Applications that do not perform inference do not incur the dependency load. ### 1.3 Session Management Sessions are the runtime objects that bind a loaded model to a specific execution provider configuration: - **Managed session pool**: Sessions are keyed by model identifier + provider configuration, pooled for reuse, and released when no longer needed. - **Full lifecycle control**: Create, run inference, release for reuse, and dispose of sessions with explicit lifecycle methods. - **Concurrent session support**: Multiple models can be loaded and actively serving inference requests simultaneously, with per-session isolation. - **Input/output validation**: Before executing inference, feed tensor names and fetch tensor names are validated against the model's declared input/output signatures. --- ## 2. Model Management A complete model lifecycle system covering discovery, acquisition, format handling, optimization, caching, and eviction. ### 2.1 Model Registry and Discovery - **Central registry operations**: Register new models with metadata, list all registered models, look up models by identifier or capability tag, and remove deprecated models. - **Model discovery**: Discover available models from configured Hugging Face Hub repositories, local directories, and custom model servers. - **Model versioning**: Track multiple versions of the same model with immutable version records that cannot be modified after registration. - **Rich metadata**: Each model record stores architecture family (LLaMA, Mistral, BERT, etc.), capability tags (chat, code, embedding, vision), license identifier, parameter count, supported context length, quantization format, and arbitrary custom fields. ### 2.2 Model Acquisition - **Download with progress**: Download models from Hugging Face Hub, GGUF repositories, or any HTTP URL with real-time progress tracking — useful for long downloads of multi-gigabyte models. - **Checksum verification**: After download, compute and verify SHA-256 and other hash algorithms against expected values to confirm file integrity before loading. - **Format auto-detection**: Automatically identify a model file's format by reading its header bytes — no file extension required. ### 2.3 Model Format Support | Format | Capability | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **SafeTensors** | Load and parse Hugging Face SafeTensors files with tensor metadata extraction. SafeTensors is the safest format — no arbitrary code execution during load. | | **GGUF** | Load GGUF (llama.cpp) quantized model files — the most common format for CPU-friendly quantized LLMs. Full header and metadata parsing. | | **GGML** | Load legacy GGML format files (the predecessor to GGUF) for backward compatibility with older quantized models. | | **PyTorch** | Load PyTorch `.pt` and `.pth` checkpoint files directly. | ### 2.4 Model Conversion and Optimization - **Cross-format conversion**: Convert models between supported formats (e.g., PyTorch → SafeTensors, SafeTensors → GGUF) for deployment optimization. - **Optimization pipeline**: Chain format detection, loading, quantization, and graph optimization in a single declarative pipeline. - **Benchmarking**: Measure inference speed (tokens/second), throughput (requests/second), and latency percentiles across different providers and configurations. - **Profiling**: Detailed execution profiling identifying which model layers consume the most time and GPU memory, enabling targeted optimization. ### 2.5 Model Caching and Lifecycle - **In-memory model cache**: Keep loaded model weights in memory to avoid reloading from disk on every request, eliminating multi-second startup latency per inference request. - **LRU and LFU eviction**: When memory pressure rises, evict the least-recently-used or least-frequently-used models first. LRU (Least Recently Used) evicts the model accessed longest ago; LFU (Least Frequently Used) evicts the model called least often overall. - **Preloading**: Configure specific models to be loaded into memory at service startup for guaranteed zero-latency first inference. - **Warm model pool**: Maintain a pool of model sessions already initialized and ready to accept requests, eliminating session initialization from the critical path. --- ## 3. Inference Pipeline High-throughput, production-grade inference pipeline with dynamic batching, attention optimization, and multi-GPU parallelism. ### 3.1 Dynamic and Continuous Batching Batching groups multiple inference requests together for simultaneous processing, making far better use of GPU parallelism than processing requests individually: - **Dynamic batching**: Accumulates incoming requests in a queue until either a configurable maximum batch size is reached or a maximum queue delay expires, then dispatches the batch. Balances throughput (larger batches) against latency (shorter queue delays). - **Continuous batching**: For autoregressive LLM generation, processes token generation across multiple in-flight sequences simultaneously without waiting for any sequence to complete. New requests join the batch as slots become available, maximizing GPU utilization. - **Model-keyed routing**: Requests for different models are batched independently, so high-traffic models do not block lower-traffic models. - **Queue management**: Configurable maximum queue depths per model with backpressure signaling and dropped-request metrics for capacity planning. ### 3.2 LLM Serving - **Full serving pipeline**: End-to-end LLM serving from HTTP request receipt through request queuing, batch formation, GPU inference, and streaming response delivery. - **Priority queuing**: Callers specify a priority level for their requests. High-priority requests (interactive user-facing) are served ahead of low-priority background batch requests. - **Throughput optimization**: Intelligent scheduling that maximizes GPU utilization by overlapping attention computation, memory transfers, and token sampling. ### 3.3 Attention Mechanisms and KV Cache The KV (key-value) cache stores intermediate attention computations so they do not need to be recomputed when generating each new token: - **KV cache management**: Explicit allocation, growth, and eviction of key-value caches scoped to each generation request. The KV cache is the primary memory consumer during generation. - **Paged attention**: Splits the KV cache into fixed-size pages (analogous to virtual memory paging), reducing memory fragmentation and allowing larger effective context windows within a fixed memory budget. - **Flash attention**: Fused GPU attention kernels that compute attention without materializing the full attention matrix in HBM (high-bandwidth memory), reducing memory bandwidth requirements by up to 5–10x for long sequences. - **Speculative decoding**: Uses a small, fast "draft" model to generate several candidate tokens speculatively, then verifies them in parallel with the large target model. Accepted speculations reduce the number of full-model forward passes needed, cutting generation latency by 2–3x for typical outputs. ### 3.4 Multi-GPU Parallelism These strategies enable models too large for a single GPU, or accelerate large models by distributing computation: - **Tensor parallelism**: Splits individual model weight matrices across multiple GPUs so each GPU holds a shard of each layer and performs its computation in parallel, then synchronizes via all-reduce operations. - **Pipeline parallelism**: Assigns consecutive transformer layers to different GPUs, pipelining computation so GPU N+1 begins computing layer N+2 while GPU N computes layer N+1 — improving overall throughput at the cost of inter-GPU communication overhead. - **Model sharding**: Automatic sharding strategy selection based on model architecture and available GPU configuration. - **Mixture-of-Experts (MoE) parallelism**: For MoE models (Mixtral, GPT-4 architecture), routes different expert sub-networks to different GPUs, allowing each GPU to specialize in a subset of experts. ### 3.5 Caching Strategies - **Prefix caching**: Cache the KV states corresponding to common prompt prefixes (system prompts, few-shot examples) so repeated requests sharing the same prefix don't recompute attention for those tokens. - **Radix attention caching**: A radix-tree-based cache structure that identifies the longest shared prefix between any incoming request and any cached prefix, maximizing cache reuse. - **Prompt result caching**: Cache complete prompt → response mappings for fully deterministic (temperature=0) queries where the same input always produces the same output. --- ## 4. Generation Control Fine-grained control over how the model samples from its output distribution during text generation. ### 4.1 Sampling Strategies - **Temperature sampling**: Scale the model's output logit distribution by a temperature value before sampling. Temperature 0 = deterministic (always pick the most probable token). Temperature 1 = unmodified distribution. Temperature > 1 = more random and creative. - **Top-k sampling**: Restrict token selection to the k most probable tokens at each step, truncating the long tail of unlikely tokens. Values of 40–100 are typical. - **Top-p (nucleus) sampling**: Dynamically restrict token selection to the smallest set of tokens whose cumulative probability mass exceeds the threshold p (typically 0.9–0.95). Adapts the candidate pool size based on the sharpness of the distribution. - **Beam search**: Generate multiple candidate sequences in parallel (beams) and select the overall most probable complete sequence. Higher beam width improves quality at proportionally higher compute cost. ### 4.2 Logit Processing - **Custom logit processor pipeline**: Chain multiple logit processors, each transforming the raw logit distribution before sampling. Processors execute in order, enabling complex composed generation policies. - **Repetition penalty**: Reduce the probability of tokens that have already appeared in the generated text, discouraging repetitive phrases and circular reasoning. - **Logit bias**: Apply per-token additive biases to directly steer generation — positive bias increases a token's probability (useful for format adherence), negative bias decreases it (useful for avoiding specific words). ### 4.3 Streaming and Stop Conditions - **Token streaming**: Deliver each generated token to the consumer as it is produced, rather than buffering the complete response. Enables real-time display of generation progress. - **Stop sequences**: Halt generation when specific token sequences (e.g., `"\n\n"`, `"###"`, a closing XML tag) appear in the output — essential for structured generation formats. - **Max token limits**: Cap total generation length at a configurable token count, preventing runaway generation that consumes excessive compute. --- ## 5. Multi-Adapter Serving Serve multiple fine-tuned model variants from a single loaded base model, dramatically reducing memory requirements compared to loading each variant as a separate model. - **Multi-LoRA serving**: LoRA (Low-Rank Adaptation — a fine-tuning technique that trains a small set of rank-decomposed weight matrices that modify specific model layers) enables serving multiple adapters concurrently from a single shared base model. Each adapter adds roughly 1–5% of the base model's memory footprint rather than a full model copy. - **Hot adapter switching**: Swap LoRA adapters in and out of active use without unloading the base model, enabling per-request adapter selection for personalization or domain specialization. - **Context length extension**: Extend a model's effective context length beyond its training context limit using RoPE (Rotary Position Embedding) scaling techniques including linear scaling, NTK-aware scaling, and dynamic NTK scaling. --- ## 6. Graph Optimization and Quantization Reduce model inference latency and memory requirements through graph-level optimization and numerical precision reduction. ### 6.1 Graph Optimization ONNX graph optimization applies algebraic simplifications and structural changes to the model computation graph: - **Optimization levels**: Apply basic (constant folding, node elimination), extended (operator fusion, layout optimization), or all available optimizations in order of aggressiveness. - **Operator fusion**: Merge consecutive compatible operators (MatMul + Add → FusedAdd, Conv + BatchNorm → FusedConvBN) into single fused operations that execute faster with fewer memory round-trips. - **Memory planning**: Optimize tensor allocation patterns using arena-based memory planning that pre-allocates a single large buffer and carves tensors from it, reducing allocation overhead and fragmentation. ### 6.2 Quantization Quantization reduces the numerical precision of model weights and/or activations, trading a small accuracy loss for significantly smaller memory footprint and faster inference: | Quantization Type | Memory Reduction | Accuracy Impact | Use Case | | ------------------------ | ---------------- | --------------- | ----------------------------------------------------------------- | | **FP16 (float16)** | ~2x | Negligible | GPU inference where FP16 hardware support is available | | **INT8 symmetric** | ~4x | Small | CPU and GPU inference for general-purpose use | | **INT8 asymmetric** | ~4x | Minimal | Better accuracy than symmetric for activations | | **INT4 (nibble-packed)** | ~8x | Moderate | Very memory-constrained devices; GGUF models on consumer hardware | | **Dynamic quantization** | ~4x | Weight-only | No calibration data needed; quantize on-the-fly at load time | | **Static quantization** | ~4x | Better accuracy | Calibration data improves value range estimation | Additional quantization capabilities: - **Calibration collection**: Run a representative dataset through the model to collect tensor value range statistics for static quantization. - **Percentile calibration**: Use percentile-based range clipping (e.g., 99.9th percentile) rather than absolute min/max to reduce the impact of outliers on the quantization range. - **Batch quantization**: Apply quantization to entire weight tensor maps in a single pass with compression ratio tracking. - **Dequantization**: Convert quantized tensors back to float32 for accuracy analysis, fine-tuning, or mixed-precision inference experiments. --- ## 7. Tensor Operations Low-level typed tensor creation and manipulation — the data layer that all higher-level Nous capabilities build on. - **Fully typed tensor creation**: Create tensors with explicit numerical types: float32, float64, int8, uint8, int16, uint16, int32, uint32, int64, uint64, and bool. Type is part of the tensor's compile-time type signature, preventing type confusion bugs. - **Immutable dimension metadata**: Tensors carry their shape dimensions as immutable metadata. Any operation that would produce a shape mismatch is detectable before execution. - **Native typed array backing**: Tensors are backed by JavaScript TypedArrays (Float32Array, Int8Array, etc.) that map directly to WASM or native memory, enabling zero-copy data exchange between JavaScript and the inference runtime. - **Pre-execution validation**: Before calling ONNX Runtime inference, all input tensors are validated for correct shape, data type, and element count against the model's declared input schema. --- ## 8. Embedding Generation Generate, process, and manage vector embeddings — dense numerical representations of text, images, and audio that capture semantic meaning in a form that allows mathematical comparison. ### 8.1 Embedding Providers - **Multi-provider registration**: Register multiple embedding providers (OpenAI `text-embedding-3-large`, Cohere `embed-english-v3.0`, local sentence-transformers models, etc.) with priority rankings. - **Provider fallback**: When the highest-priority provider returns an error or exceeds latency thresholds, the next provider is tried automatically. - **Per-provider statistics**: Track success rate, failure rate, average latency, and timeout count per provider for data-driven routing policy decisions. - **Configurable timeouts**: Per-request timeout limits with provider-level timeout tracking; slow providers are deprioritized based on observed latency statistics. ### 8.2 Modality Support | Modality | Capability | | -------------------------- | ------------------------------------------------------------------------------------------------------------ | | **Text embeddings** | Dense vector representations of text — the foundation for semantic search, clustering, and RAG | | **Image embeddings** | Vector representations of images for visual similarity search, duplicate detection, and image classification | | **Audio embeddings** | Vector representations of audio segments for speaker identification, acoustic similarity, and audio search | | **Multi-modal embeddings** | Combined embeddings from text + image pairs using models like CLIP, enabling cross-modal similarity | ### 8.3 Embedding Processing - **Normalization**: Normalize embedding vectors to unit L2 length so that cosine similarity equals dot product — a standard preprocessing step for similarity search. - **Dimensionality reduction**: Reduce embedding dimensions using PCA or random projection while preserving semantic structure — useful when downstream systems have dimension limits. - **Output dimension control**: Truncate embeddings to a requested dimension count (e.g., 1536-dim to 512-dim) for storage and compute savings with controlled quality degradation. - **Batch embedding**: Generate embeddings for multiple inputs in a single model call, reducing overhead and improving throughput for bulk operations. - **Streaming embedding**: Process large input sets in a streaming fashion, yielding embeddings as they are generated rather than buffering everything in memory. - **Embedding caching**: Cache generated embeddings keyed by input hash to avoid recomputing embeddings for inputs that have already been processed. ### 8.4 Embedding Compression - **Binary embeddings**: Quantize float embeddings to binary (1-bit) representations. Binary embeddings are 32x smaller than float32 and can be compared with bitwise XOR + popcount for ultra-fast approximate search. - **Matryoshka embeddings**: Models trained with MRL (Matryoshka Representation Learning) produce embeddings where any prefix of the full vector is independently semantically meaningful, allowing truncation to shorter dimensions without model retraining. - **General compression**: Apply additional lossless or lossy compression to embedding vectors for cold storage efficiency. --- ## 9. Similarity Search Search and rank documents or data points by vector similarity — the core operation underlying semantic search, RAG, recommendation systems, and duplicate detection. ### 9.1 Nearest Neighbor Search - **Approximate Nearest Neighbor (ANN)**: Fast approximate search using HNSW (Hierarchical Navigable Small World — a graph-based index that navigates from coarse to fine neighbors) and IVF (Inverted File — a clustering-based index) structures. Enables sub-linear search times on billion-scale datasets — finding the 10 closest vectors out of 1 billion in milliseconds. - **Exact nearest neighbor**: Brute-force linear scan through all vectors for mathematically exact results. Practical for datasets up to a few hundred thousand vectors where maximum accuracy matters more than speed. - **Hybrid search**: Combine vector similarity (ANN) with keyword matching (BM25 — a probabilistic ranking function that scores documents by term frequency and inverse document frequency) and merge the ranked lists using Reciprocal Rank Fusion for better overall relevance than either approach alone. ### 9.2 Distance Metrics | Metric | Formula Intuition | Best Use Case | | ---------------------- | ------------------------------------------------ | ---------------------------------------------------- | | **Cosine similarity** | Angle between vectors; ignores magnitude | Text embeddings where direction encodes meaning | | **Euclidean distance** | Straight-line distance in embedding space | Image embeddings where absolute position matters | | **Dot product** | Magnitude-sensitive version of cosine similarity | Embeddings from models trained with dot product loss | | **Manhattan distance** | Sum of absolute coordinate differences (L1 norm) | Robust search in high-dimensional sparse spaces | ### 9.3 Reranking Reranking takes an initial candidate list from ANN search and re-scores it with a more accurate (but more expensive) model: - **Score-based reranking**: Use a cross-encoder or scoring model to evaluate each retrieved candidate against the query and reorder by score. - **Cross-encoder reranking**: Concatenate the query and each candidate document as a pair and pass through a BERT-style cross-encoder that outputs a single relevance score. Cross-encoders outperform bi-encoder (embedding) models on relevance but are too expensive for full-corpus search — hence the two-stage retrieve-then-rerank pipeline. --- ## 10. LLM Completion APIs High-level interfaces that make it easy to use local LLMs for text generation, conversation, and structured output without managing inference sessions directly. ### 10.1 Completion Interfaces - **Text completion**: Generate a completion of an open-ended text prompt — the foundational LLM capability. - **Chat completion**: Multi-turn conversation with the standard `system`, `user`, and `assistant` message roles. System messages set persistent behavioral instructions; user/assistant turns interleave the conversation. - **Instruction following**: Generate outputs that follow explicit natural language instructions, leveraging instruction-tuned model variants. - **Multi-turn context management**: Maintain full conversation history across multiple exchanges with configurable context truncation strategies (sliding window, summarization, importance-based pruning) when the conversation exceeds the model's context limit. - **System prompts**: Define persistent system-level instructions that apply to every response in a conversation without consuming visible conversational context space. ### 10.2 Function Calling and Tool Use Function calling allows LLMs to request specific tool invocations rather than attempting to simulate them in text: - **Function declaration**: Declare callable functions with typed parameter schemas (JSON Schema format), descriptions, and required/optional parameter annotations. - **Tool use framework**: Define tools with human-readable descriptions and parameter schemas. The LLM inspects the tool list and generates structured tool-call requests when a tool would help answer the current prompt. - **Tool selection**: Context-aware selection of the appropriate tool from a registered tool list, generating structured JSON specifying the tool name and argument values. ### 10.3 Structured Output Force the model to produce output in a specific format: - **JSON mode**: Constrain all output to syntactically valid JSON, preventing free-form text contamination. - **Structured output against schema**: Generate output that strictly conforms to a specified JSON Schema, including correct types, required fields, and value constraints. - **Grammar-constrained generation**: Constrain generation to strings matching a formal context-free grammar (BNF or EBNF format) — useful for generating valid code, SQL, or domain-specific languages. - **Regex-constrained generation**: Constrain generation to strings matching a regular expression — useful for phone numbers, dates, identifiers, and other structured formats. - **Post-generation validation**: Validate generated structured output against its schema after generation, flagging any constraint violations. ### 10.4 Response Processing - **Response formatting**: Normalize raw LLM output by trimming whitespace, standardizing punctuation, and applying consistent formatting. - **Markdown rendering**: Convert LLM Markdown output to clean HTML or structured document objects. - **Code extraction**: Detect and extract code blocks (fenced with language identifiers) from mixed text/code outputs. - **Citation extraction**: Identify and extract citation references from LLM outputs — numbered references, inline citations, and bibliography-style references. - **Fact extraction**: Identify and extract factual claims and assertions from LLM outputs as discrete structured items for downstream verification or storage. --- ## 11. Natural Language Processing Built-in NLP capabilities that run locally without external API calls. - **Entity extraction**: Identify and extract named entities from text including people, organizations, locations, dates, monetary amounts, percentages, and custom domain-specific entity types. Returns entity spans, types, and confidence scores. - **Sentiment analysis**: Classify the overall emotional tone of text into categories (positive, negative, neutral, mixed) with confidence scores. Supports both document-level and sentence-level sentiment. - **Text classification**: Classify text into predefined or zero-shot categories using both fine-tuned classification heads and zero-shot prompting with generative models. Zero-shot classification uses an LLM to categorize text without training examples for that specific category. --- ## 12. Prompt Engineering A comprehensive prompt template system with variable substitution, conditional logic, loops, versioning, testing, optimization, and few-shot example management. ### 12.1 Template System Nous uses a Handlebars-inspired template syntax for prompt construction: - **Variable substitution**: Insert values using `{{variable}}` syntax. Access nested object properties with dot notation: `{{user.profile.name}}`, `{{config.model.temperature}}`. - **Conditional sections**: Include or exclude prompt sections based on variable values using `{{#if condition}}`, `{{else}}`, and `{{/if}}`. Conditions support `==`, `!=`, and `!` negation. - **Loop constructs**: Iterate over arrays with `{{#each items as item}}...{{/each}}`, with automatic index tracking via `{{@index}}` for numbered lists. - **Strict mode**: When enabled, any template variable that is not provided in the input throws an error rather than leaving a placeholder in the output — catches missing inputs before they produce confusing prompts. - **Missing value defaults**: Provide a fallback string for any unresolved variable: `{{variable | default "not specified"}}`. - **Object serialization**: Complex objects passed as template variables are automatically serialized to a human-readable JSON representation. ### 12.2 Template Management - **Template registry**: Register reusable prompt templates with unique IDs, human-readable names, descriptions, tags, and metadata. - **Template versioning**: Every modification to a registered template creates a new version record. The full revision history is queryable for audit, debugging, and rollback. - **Template search**: List and filter templates by tag, description keyword, or creation date. - **Variable auto-extraction**: Automatically detect all variables referenced in a template and return their names — useful for generating input forms or documentation. ### 12.3 Prompt Versioning and Testing - **Prompt versioning**: Version and track prompt changes over time. Tagged versions can be pinned in production deployments for reproducibility — a generation pipeline can reference a specific prompt version to ensure identical behavior across deployments. - **Prompt testing**: Define test cases with expected outputs and run automated evaluations to detect prompt regressions when templates are modified. - **Prompt optimization**: Systematic search over prompt variations (wording, ordering, few-shot examples) to find the formulation that maximizes performance on a test set. ### 12.4 Few-Shot Example Management - **Few-shot example collections**: Manage collections of input-output example pairs for in-context learning — the technique of providing examples in the prompt to guide model behavior. - **Example selection**: Given a query, select the most relevant examples from the collection using embedding similarity rather than random selection, increasing few-shot effectiveness. - **Dynamic example generation**: Generate or retrieve examples dynamically based on the current input context, enabling adaptive few-shot prompting tailored to the specific task instance. --- ## 13. Advanced Reasoning Structured reasoning strategies that improve the quality and reliability of LLM outputs for complex, multi-step problems. - **Chain-of-thought (CoT)**: Guide the model through explicit step-by-step reasoning before reaching a final answer. CoT significantly improves accuracy on arithmetic, logical, and multi-step reasoning tasks by preventing the model from jumping to conclusions. - **Tree-of-thought (ToT)**: Explore multiple reasoning branches simultaneously. The model generates several candidate next steps, evaluates each, and expands the most promising ones — like a search tree over the reasoning space. - **Graph-of-thought (GoT)**: Represent reasoning as a directed graph where nodes are thoughts and edges represent derivation relationships. Nodes can be merged, allowing insights from different reasoning paths to combine. - **Self-consistency sampling**: Generate multiple independent reasoning chains for the same problem and select the final answer by majority vote. Substantially improves reliability by reducing the variance of any single chain. - **Reflection prompting**: After the model produces an initial answer, prompt it to critically review its own reasoning, identify potential errors, and revise the answer if needed. - **Critique prompting**: A specialized form of reflection where the model generates a formal critique of its reasoning — listing specific weaknesses, logical gaps, or unsupported assumptions — before producing a revised answer. --- ## 14. Persona and Context Management Control LLM behavior by defining named personas and injecting external knowledge into prompts. - **Persona management**: Define named personas — "TechnicalAdvisor", "FriendlyHelper", "RegulatoryExpert" — each with a system prompt configuration, expertise profile, communication style, and knowledge scope. Switch between personas at conversation start or mid-session. - **Context injection**: Inject external context documents (company policies, product specifications, user profile data) into prompts as structured sections without manually concatenating strings. - **Memory injection**: Inject retrieved conversation memories and user history into prompts in a structured format that the model can reference without confusing historical context with the current exchange. - **Retrieval injection**: Inject retrieved document chunks from vector search into prompts in a format that clearly attributes source documents, enabling the model to cite them correctly. --- ## 15. Agent Architecture Build autonomous AI agents that plan, use tools, maintain memory, and coordinate with other agents to complete complex multi-step goals. ### 15.1 Agent Lifecycle - **Agent registration**: Register agents with unique identifiers, human-readable names, capability tags, tool permissions, and module configurations (which planning, memory, and tool modules to use). - **Lifecycle states**: Agents move through stopped, idle, running, paused, and error states with defined transitions. State changes trigger configurable callbacks. - **Concurrency control**: Per-agent configurable maximum concurrent task count prevents resource exhaustion from task floods. - **Graceful lifecycle management**: Start, stop, pause, and resume operations handle in-progress tasks gracefully — pausing waits for the current task step to complete before suspending. ### 15.2 Task Execution - **Task submission**: Submit tasks with a goal description, optional payload data, task-specific context, priority level (1–10), and timeout duration. - **Priority-based scheduling**: The task queue processes high-priority tasks first. Tasks with equal priority are processed FIFO. - **Timeout enforcement**: Tasks that exceed their configured timeout are automatically terminated and reported as timed-out rather than running indefinitely. - **State tracking**: Tasks progress through queued → running → completed (or failed, cancelled, timeout). Each state transition is timestamped and logged. ### 15.3 Planning and Decomposition - **Planning module**: Before executing a complex goal, the agent generates a plan — a summary, an ordered list of sub-steps, and a confidence score. Users can review the plan before execution begins. - **Goal decomposition**: Break high-level goals ("research competitors and write a comparison report") into concrete, sequenced sub-tasks with dependencies. - **Multi-step reasoning**: Chain multiple reasoning steps, preserving intermediate results between steps and using them as context for subsequent steps. - **Backtracking**: When a reasoning path reaches a dead end or produces an error, the agent can revert to an earlier decision point and try an alternative approach. - **Error recovery**: Detect errors in mid-task execution, classify them as recoverable or unrecoverable, and apply appropriate recovery strategies (retry, alternate approach, or graceful failure). ### 15.4 Agent Memory Agents maintain their own memory stores separate from conversation-level memory: - **Session state**: Persist key-value state across tasks within an agent's active session. - **Short-term memory**: Recent context from the last N interactions, providing continuity for ongoing conversations. - **Long-term memory**: Important facts and outcomes from past sessions persisted to durable storage. - **Episodic memory**: Specific significant interaction episodes — successful task completions, important decisions. - **Semantic memory**: Factual knowledge stored as vector embeddings for semantic retrieval. - **Working memory**: The active reasoning context maintained during a single task execution step. ### 15.5 Multi-Agent Systems - **Agent communication**: Agents send typed messages to other agents with structured payloads, enabling collaborative problem-solving across specialists. - **Multi-agent coordination**: Coordinate groups of agents working on related sub-tasks with dependency management and synchronized completion. - **Task delegation**: One agent delegates a specialized sub-task to another agent best equipped for that domain, collecting the result asynchronously. - **Agent supervision**: A supervisor agent oversees other agents' execution, monitoring for quality issues, providing corrective instructions, and aggregating outputs. ### 15.6 Runtime Events and Observability - **Event types**: Agent registered, unregistered, started, stopped; task submitted, started, completed, failed, timed out, cancelled; module-level events for planning, tool execution, and memory operations. - **Event history**: Complete timestamped event log for each agent, queryable for debugging and analytics. - **Runtime statistics**: Real-time metrics — active agent count, queued task count, running task count, completed/failed/timeout counts, average task latency, and event counts. --- ## 16. Document Processing and RAG Ingest documents from raw files into searchable vector indexes for RAG (Retrieval-Augmented Generation — the technique of retrieving relevant document passages and injecting them into the LLM's context window to ground responses in specific knowledge). ### 16.1 Document Ingestion - **End-to-end pipeline**: A single pipeline accepts a raw document file and produces indexed, searchable chunks — handling format detection, text extraction, cleaning, chunking, embedding, and indexing. - **PDF parsing**: Extract text content, headings, tables, and structural metadata from PDF files, including PDFs with complex layouts. - **HTML parsing**: Parse web pages and extract clean text, stripping navigation, advertisements, and boilerplate while preserving article content. - **Markdown parsing**: Parse Markdown documents preserving heading hierarchy, code block boundaries, and list structure as metadata for semantically-aware chunking. - **Code file parsing**: Parse source code files with language-aware structure extraction — identifying functions, classes, imports, and docstrings as structural units. ### 16.2 Content Extraction - **Table extraction**: Detect and extract tabular data from documents, preserving column headers and row relationships for structured data retrieval. - **Image extraction**: Extract embedded images from documents for separate vision analysis or attachment to document chunks. - **Metadata extraction**: Extract document-level metadata — title, author, creation date, modification date, keywords, and document properties. - **Link extraction**: Extract hyperlinks and cross-references for knowledge graph construction or link analysis. ### 16.3 Chunking Strategies Chunking divides documents into pieces appropriately sized for the LLM's context window while preserving coherent meaning: - **Semantic chunking**: Split at natural semantic boundaries (end of a complete thought, paragraph, section) rather than at arbitrary token counts. Produces more coherent chunks. - **Sentence chunking**: Split at grammatical sentence boundaries, grouping multiple sentences until a target token count is reached. - **Token chunking**: Split at token count boundaries with configurable overlap between adjacent chunks, ensuring no text segment falls in a gap between chunks. - **Recursive chunking**: Apply multiple splitting strategies hierarchically — split by heading, then by paragraph, then by sentence — to produce chunks at the most appropriate granularity for each document section. - **Configurable parameters**: Chunk size (in tokens or characters), overlap ratio, minimum chunk size, and splitting strategy are all configurable per ingestion pipeline. ### 16.4 Vector Indexing - **Vector indexing**: Convert document chunks to embedding vectors and store them in an HNSW or IVF index for approximate nearest-neighbor retrieval. - **Hybrid indexing**: Build both a vector index and a BM25 keyword index for the same document collection, enabling hybrid search that combines both retrieval signals. - **Incremental indexing**: Add, update, or remove individual documents without rebuilding the entire index — essential for knowledge bases that are continuously updated. - **Index optimization**: Rebuild or optimize index structures periodically to maintain query performance as the index grows. ### 16.5 Context Retrieval - **Semantic context retrieval**: Given a natural language query, retrieve the top-K most semantically relevant document chunks from the vector index. - **Multi-source retrieval**: Query multiple separate indexes (e.g., a product documentation index and a support ticket history index) and merge their results with configurable weighting. - **Relevance scoring**: Score retrieved chunks by their semantic similarity to the query and apply a minimum threshold to filter out irrelevant results. --- ## 17. Computer Vision — `@nous/vision` Local computer vision inference covering detection, recognition, scene understanding, and generative image capabilities. All models run locally — no cloud vision API calls required. ### 17.1 Object Detection and Classification - **Object detection**: Detect and locate objects in images, returning bounding box coordinates, class labels, and confidence scores. Supports standard models (YOLOv8, DETR, etc.). - **Image classification**: Classify images into categories using models like EfficientNet, ViT, and custom-trained classifiers. - **Instance segmentation**: Generate pixel-level masks for each detected object instance, distinguishing between separate instances of the same class (individual people in a crowd). - **Semantic segmentation**: Assign a class label to every pixel in an image — every pixel is classified as road, sidewalk, sky, building, etc. - **Panoptic segmentation**: Combine instance and semantic segmentation: "stuff" classes (road, sky) get semantic labels while "things" classes (cars, people) get individual instance masks. ### 17.2 Facial Analysis - **Facial recognition**: Detect faces and generate 128–512 dimensional facial identity embeddings for matching and identification. - **Face landmark detection**: Detect 68–478 facial landmark points (eyes, nose, mouth, jaw, eyebrows) for alignment, expression transfer, and face analysis. - **Face restoration**: Enhance degraded, blurry, or low-resolution face images using models like GFPGAN and CodeFormer. - **Emotion recognition**: Classify facial expressions into primary emotion categories (happiness, sadness, anger, fear, disgust, surprise, neutral) with continuous probability scores. - **Age and gender estimation**: Estimate age range and gender presentation from facial features using regression and classification models. ### 17.3 Document and Text Recognition (OCR) - **OCR engine**: Extract text from images of documents, screenshots, and photos using Tesseract, PaddleOCR, or deep learning OCR models. - **Handwriting recognition**: Recognize and transcribe handwritten text, including cursive and mixed print/cursive styles. - **Document layout analysis**: Detect and classify document regions — headers, body text, captions, footnotes, columns, page numbers — to understand document structure before text extraction. - **Table recognition**: Detect tabular structures in document images and extract cell contents with their row/column positions as structured data. - **Barcode and QR code decoding**: Detect and decode 1D barcodes (Code 128, EAN), QR codes, and Data Matrix codes from images. - **License plate recognition**: Detect and read vehicle license plates from road camera or parking lot images. ### 17.4 Scene Understanding - **Depth estimation**: Estimate per-pixel depth from a single monocular image using models like MiDaS and DPT — no stereo camera or depth sensor required. - **Surface normal estimation**: Estimate the surface normal direction at each pixel, indicating the 3D orientation of surfaces. - **Human pose estimation**: Detect human body keypoint positions (shoulders, elbows, wrists, hips, knees, ankles) from images or video frames. - **Visual question answering (VQA)**: Answer natural language questions about image content: "What color is the car?", "How many people are in this image?". - **Image captioning**: Generate natural language descriptions of image content, from brief ("A dog running in a park") to detailed multi-sentence descriptions. ### 17.5 Image Generation — Diffusion Models Nous supports running diffusion models locally for private, zero-cost-per-generation image generation: - **Stable Diffusion (1.5, 2.0, 2.1)**: Text-to-image generation using standard SD model weights. - **Stable Diffusion XL (SDXL)**: Higher-resolution, higher-quality text-to-image generation using the SDXL architecture. - **Stable Diffusion 3 (SD3)**: Latest-generation text-to-image generation with improved text rendering and compositional understanding. - **Flux models**: Support for the Flux model architecture (Black Forest Labs) for text-to-image generation. - **Diffusion model framework**: A general pipeline framework for loading and running any diffusion model that conforms to the standard checkpoint format. ### 17.6 Image Editing and Transformation - **Image-to-image**: Transform an existing image using a diffusion model guided by a text prompt and a strength parameter — higher strength means more transformation relative to the original. - **Inpainting**: Fill in masked regions of an image with contextually appropriate generated content — remove objects, repair damage, or complete partially visible elements. - **Outpainting**: Extend an image beyond its original canvas boundaries by generating coherent continuation of the scene. - **Background removal**: Remove the background from an image using salient object detection, producing a transparent-background cutout of the foreground subject. - **Style transfer**: Apply the artistic style (brushstroke texture, color palette, level of abstraction) from a reference style image to the content of a source image. - **Image composition**: Composite multiple images together with seamless blending at boundaries and alignment correction. - **ESRGAN super-resolution upscaling**: Upscale low-resolution images to 2x, 4x, or 8x resolution using Real-ESRGAN or ESRGAN generative super-resolution models. - **Variation generation**: Generate multiple variations of an existing image preserving its structure and composition while varying style, details, and appearance. --- ## 18. Model Training and Fine-Tuning — `@nous/training` Training capabilities that let an application adapt a base model to a domain without leaving the sovereign, local infrastructure. All training runs on the same hardware abstraction (Section 1) as inference. ### 18.1 Fine-Tuning Strategies - **Full-parameter fine-tuning**: Update every weight of the base model. Highest capacity to adapt, highest cost — requires optimizer state and gradients for the full parameter set, so VRAM demand is several times the inference footprint. - **LoRA (Low-Rank Adaptation)**: Freeze the base weights and train only a small set of rank-decomposed update matrices injected into selected layers. The trained adapter is roughly 1–5% of the base model's size, and multiple LoRA adapters from one base can be served concurrently via the multi-adapter path (Section 5). - **QLoRA**: LoRA applied on top of a base model held in 4-bit quantized form. Backpropagation flows through the quantized weights into the float adapter, cutting training VRAM enough to fine-tune a large model on a single consumer GPU. - **Adapter rank and target-layer selection**: The LoRA rank and the set of target modules (attention projections, MLP layers) are configurable, trading adapter capacity against size and training cost. ### 18.2 Preference and Reinforcement Tuning - **RLHF (Reinforcement Learning from Human Feedback)**: Train a reward model from human preference comparisons, then optimize the policy model against that reward with a reinforcement objective, while a KL penalty against the reference model prevents the policy from drifting into degenerate text. - **DPO (Direct Preference Optimization)**: Optimize directly on pairs of preferred and rejected responses with a closed-form preference loss, skipping the separate reward model and RL loop. Cheaper and more stable than RLHF for most preference-alignment tasks. - **Preference dataset handling**: Preference pairs are ingested, validated for schema completeness, and split into train and evaluation partitions; an incomplete pair (missing the rejected response) is rejected at ingestion. ### 18.3 V2 Anti-Cheat Classifier Training Loop `@nous/training` owns a concrete classifier-training loop for the V2 fighting- game project. It trains four classifiers — smurf detection, win-trading, coordinated-throw, and geographic-anomaly — from labeled examples that V2 publishes on `@oshun/event-bus` topics (clean examples and positive examples per classifier). - **Plan validation**: `buildNousV2AntiCheatTrainingPlan` is the contract governing a training run. It rejects the plan if any of the four classifiers, its event topics, or its label coverage is incomplete — a classifier cannot be trained on partial supervision. - **Review-only output**: The trained classifiers are review-only. Their output may only prioritize a manual fair-play review queue; automatic discipline is not permitted, and the classifier path is off the deterministic rollback path so it can never affect gameplay simulation. - **Composition boundary**: `@v2/nous-anti-cheat-classifiers` composes this training contract with the paired Model Card (Section 21); it explicitly replaces any prior reference to a non-existent `@nous/ml-pipeline`. --- ## 19. Model Evaluation — `@nous/autoresearch-evals` Evaluation harnesses that measure model and agent quality against fixed benchmarks, so a fine-tuned or newly registered model is promoted on evidence. - **Benchmark harnesses**: Run a model against a benchmark suite — a fixed set of tasks with known-correct answers — and report per-task and aggregate scores. Suites cover reasoning, code, and knowledge tasks. - **Pairwise model comparison**: Evaluate two model versions on the same suite under identical decoding settings and report the win rate, giving the A/B promotion decision (Section 2) a measured basis. - **Agent benchmarks**: Evaluate the agent runtime (Section 15) end to end on multi-step tasks, scoring task completion rather than single-response quality. - **Regression gating**: An evaluation run can act as a release gate — a model whose score regresses against the incumbent on a designated suite is blocked from promotion. - **Reproducible runs**: Each evaluation records the model revision, decoding configuration, suite version, and seed, so a score can be reproduced and audited later. --- ## 20. MLOps, Serving, and Infrastructure Operational primitives for deploying and running models across the hardware spectrum, from a multi-GPU server to a serverless edge function. ### 20.1 Model Serving Lifecycle - **Versioned deployment**: A model version from the registry (Section 2) is deployed to a serving slot; deployments are versioned so a known-good version can be restored. - **Warm pools and preloading**: Serving slots draw from the warm session pool (Section 2) so the first request after a deployment does not pay cold-start latency. - **Health and capacity signals**: Per-model queue depth, batch statistics, and dropped-request counts (Section 3) feed capacity planning and autoscaling decisions. ### 20.2 Edge and Serverless Deployment - **Quantized edge models**: For serverless and edge targets, models are served in INT8 or INT4 form (Section 6) so the deployment artifact and memory footprint fit within constrained environments. - **Browser deployment**: The `webgpu` provider runs models in-browser via `onnxruntime-web`; `onnxruntime-node` is an optional dependency, so a package can be imported and compiled in a browser or serverless target that cannot install the native runtime, with the runtime loaded lazily only at first inference. - **CPU fallback**: Any environment without a GPU falls back to the universal `cpu` provider, so a deployment never hard-fails for lack of acceleration. ### 20.3 GPU Infrastructure - **Multi-GPU placement**: Tensor, pipeline, and MoE parallelism (Section 3) place a model too large for one GPU across several, with the sharding strategy selected from the model architecture and the available GPU topology. - **Memory sizing**: Deployment is sized from documented VRAM requirements — for example a 7B model needs roughly 14 GB at FP16 or 4 GB at INT4, and a 70B model at INT4 needs roughly 40 GB, served via multi-GPU tensor parallelism. - **GPU autoscaling for world-model serving**: The real-time world-model serving layer (Section 22) scales GPU allocation with session load. --- ## 21. Model Safety and Governance — `@nous/safety` Safety metadata and dual-use controls for the models Nous serves. `@nous/safety` does not own product behavior or player settings — it hosts the records that make a deployed model accountable. ### 21.1 Model Card Hosting `@nous/safety` hosts AI Model Card metadata for the V2 shipping AI systems: the Adaptive AI Director, AI commentary, the anti-cheat classifier suite, and the generation pipelines. Each Model Card is built by a dedicated contract (`buildV2AdaptiveAIModelCard`, `buildV2AICommentaryModelCard`, `buildV2AntiCheatClassifierModelCard`, `buildV2GenerationPipelinesModelCard`) and records: - The limited-risk classification of the system. - The disclosure obligations and the fallback behavior when the AI is unavailable. - The evaluation results backing the system. - The source-package list and the hosted document path. The Model Card is paired with `@themis/accountability` for the AI-system-of- record and conformity-audit export. Boundaries are explicit: `@nous/safety` does not own player settings or rollback behavior, `@psyche/action-safety` owns Adaptive AI classifier transparency, and `@v2/eu-ai-act-surface` composes the overall V2 shipping AI registry. ### 21.2 Dual-Use Controls - **Review-only AI outputs**: Safety-relevant classifiers (anti-cheat) are constrained to review-only, appealable, off-rollback operation — their output prioritizes human review and never triggers automatic enforcement. - **Dual-use classification for autonomous research**: The autonomous-research substrate (Section 22) carries dual-use classifiers, self-amplification guards, and budget governors so a self-improving research loop cannot escalate unchecked. - **Accountability reporting**: Safety records are exportable as accountability reports for conformity audit, keeping a deployed model traceable to its evaluation evidence and disclosure record. --- ## 22. Research and Cooperative Intelligence Extensions TODO Phases 176-179 extend Nous beyond the Phase 85-98 ML sovereignty core with four planned capability layers. All four are owned at the model, search, training, evaluation, and optimization layer; product decisions, domain policy, and settlement authority remain with the consuming domains. The features below are planned and specified against their type contracts. ### 22.1 Neural World Models and Latent Dynamics World models predict how an environment evolves under an agent's actions in a learned latent space, so an agent can plan by imagining trajectories instead of acting in the real environment. The reusable abstraction is `WorldModel` with four operations: - **`encode(observation)`**: Maps a raw observation to a latent state. Latent state is one of four families — a deterministic vector, a stochastic state (mean and log-variance), a categorical latent (grouped logits), or a token latent (codebook indices) — selected per model architecture. - **`step(state, action)`**: Advances the latent state by one action and returns the next state, a predicted reward, a continuation probability, and optionally a predicted next observation. Actions span continuous, discrete, multidiscrete, and multimodal-token action spaces. - **`imagine(state, policy, horizon)`**: Rolls the policy forward inside the model for `horizon` steps, producing an imagined trajectory used for planning and policy training without environment interaction. - **`getUncertainty(state, action)`**: Returns epistemic uncertainty for a state-action pair, so a planner can distinguish a confident prediction from an extrapolation into unseen dynamics. Planned model families implementing the abstraction: JEPA-family predictive encoders (I-JEPA, V-JEPA, Audio-JEPA, multimodal JEPA); DreamerV3/RSSM and the MuZero family (MuZero, EfficientZero, Stochastic MuZero) for model-based RL; Genie-class action-conditioned video worlds and DIAMOND/GameNGen/Oasis diffusion dynamics. Replay buffers and trajectory datasets are populated from Phase 85 flywheel outputs — Maya sessions, Galatea teleoperation, Psyche computer-use. Trained world models drive Galatea robotic imagination, Hathor NPC planning, and Iris tool-action look-ahead. A real-time Rust serving layer provides world-session persistence, multi-player shared latent sessions, and GPU autoscaling, scored by world-model benchmarks and human-factors evaluation. The `@nous/world-model-*` package family carries the core abstraction, the model families, and the serving layer. Consumers own their side of the contract: Maya interactive neural universes, Galatea VLA world-model fusion, Hathor NPC imagination, Iris look-ahead planner, and Metis/Kuanyin as look-ahead and safety consumers. ### 22.2 Interpretability and Continual Learning Mechanistic interpretability and lifelong-learning primitives over every Oshun-served LLM or vision model. - **Hooked models**: A `HookedModel` wraps a served model with TransformerLens-compatible activation hooks at fixed taps — `resid_pre`, `resid_post`, `mlp_pre`, `mlp_post`, the four attention projections (`attn_q`, `attn_k`, `attn_v`, `attn_o`), and `logits`. `capture()` records activations at the configured hooks into a Zarr store for later analysis. - **Attribution**: Activation statistics, attribution patching, integrated gradients, SmoothGrad, and LIME quantify which inputs and internal components drive a given output. - **Sparse autoencoders**: JumpReLU, TopK, and gated SAEs decompose dense activations into interpretable features, with feature autointerpretation, feature splitting, ablation, steering, and concept-dictionary export. - **Circuit discovery**: ACDC, EAP-IG, transcoders, and feature-circuit visualizers isolate the sub-network responsible for a behavior; planned investigations target refusal, factual recall, code, and deception circuits, with safety reports as output. - **Probes**: Linear probes, RSA, CKA, TCAV, and emergent-capability probes test for specific represented concepts and produce model cards. - **Continual learning**: A `ContinualTaskStream` yields a sequence of tasks in one of four modes — task-incremental, class-incremental, domain-incremental, or blurry — each task carrying its dataset and an evaluation function. Forgetting metrics and CL benchmarks measure catastrophic forgetting across the stream. Mitigations cover regularization (EWC, Online EWC, Synaptic Intelligence, MAS), architecture (progressive networks, DEN, PackNet, supermasks), and rehearsal (CLEAR, GEM/A-GEM, DER++, generative and privacy-preserving replay), plus policy distillation and Kickstarting. - **Integration**: SAE side-channel signals, circuit-stability metrics, and forgetting watchdogs feed Kuanyin safety and Galatea fleet learning as release gates. The `@nous/interp-*` and `@nous/continual-*` package families carry the interpretability and continual-learning surfaces; co-owner consumers are Kuanyin (safety reports and gating), Galatea (fleet continual learning), Iris/Veritas (model transparency), and Sophia/Metis/Mnemosyne (probing of their served models). Interpretability-driven safety reports are the AISI reporting surface named in Phase 177. ### 22.3 Autonomous Research and Agentic Scientist A substrate for autonomous research agents that read literature, form conjectures, run experiments, and report results. - **Literature graph**: Adapters for Semantic Scholar, OpenAlex, arXiv, Crossref, PubMed, CORE, bioRxiv, and medRxiv build a citation graph supporting citation traversal, novelty scoring, prior-art detection, and method/dataset/claim extraction. Survey and deep-research agents run on the graph with citation-quality enforcement. - **Search and reasoning kernels**: A unified `SearchKernel` exposes `expand`, `evaluate`, `select`, and a budgeted `run` — the common interface behind LATS, Tree-of-Thoughts, Graph-of-Thoughts, rStar-Math, and RAP. Process reward models and outcome reward models score intermediate and final states, with reward-drift monitors guarding against reward hacking. - **Self-improvement loops**: Absolute Zero Reasoner, STaR/V-STaR, self- and meta-rewarding loops, SPIRAL, Voyager-class skill libraries, automatic curricula, and agent-graph evolution. - **ML and code agents**: AIDE-class ML-engineering agents and SWE-agent/SWE-ReX code agents, evaluated on MLE-Bench, RE-Bench, SWE-Bench, SWE-Lancer, and SWE-EVO, plus GPU-kernel autotuning. - **Experiment ledger**: Every trial is recorded as an `AutoresearchLedgerEntry` — trial ID and parent, agent and model revisions, code-diff hash, seed, environment hash, command, the metric before and after with its delta, an accepted flag, and compute cost (GPU-hours, dollars, wall-clock). The ledger makes a self-improvement run fully reconstructable. - **Discovery and authoring**: AlphaEvolve/FunSearch evolutionary search, async researcher pools, paper authoring and review, reproducibility bundles, and physical-lab drivers (Coscientist/A-Lab/ChemCrow patterns). - **Governance**: Autonomous-research benchmarks, observability, budget governors, self-amplification guards, dual-use classifiers, and accountability reports bound what a self-directed research loop may do. ### 22.4 Cooperative Bargaining Primitives Reusable preference-learning and optimization packages consumed by the Concordia domain (`DOMAINS/concordia/*`). Nous owns the primitives; legal, community, procurement, governance, and settlement policy remain with Themis, Kuanyin, Maat, Aje, Contracts, OpenAPI, Proto, Sophia, Oshun, Iris, and Shared. - **Preference inference**: From a `PairwisePreferenceQuery` — a case, a party, two candidates, and a reference to that party's sealed private context — the engine fits a calibrated `UtilityEstimate` per candidate: a posterior mean, a credible interval, the comparison count, the nearest known comparison, and instability warnings. When the engine should not produce a value it returns an explicit `abstention` reason — `low_confidence`, `redline_blocked`, `refused`, or `non_tradeable` — rather than a fabricated number. - **Agreement search**: An `AgreementSearchKernel` runs one of nine strategies — Nash-product genetic search, NSGA-II, MAP-Elites, MCTS/LATS, CP-SAT, MILP, Bayesian optimization, PSRO, and coalition search — over the joint outcome space, with opponent modeling, coalition-stability checks, Shapley attribution, and side-payment search. - **Sealed memory** (`@nous/concordia-sealed-memory`): Stores each party's private intake context as a sealed artifact addressed only by reference. The preference and search kernels operate on the reference; the raw private context is never exposed to the opposing party or to the dispute surface, which is what makes party-isolated bargaining possible. - **V2 deployment boundary**: `@v2/concordia-substrate` is the V2 deployment wrapper for these primitives until a standalone `@concordia/*` exists. It wires `@nous/cooperative-bargaining`, `@nous/preference-inference`, `@nous/agreement-search`, and `@nous/concordia-sealed-memory` for anti-cheat appeals, tournament-result disputes, and crew conflicts — building Concordia case sessions, sealed private-intake artifacts, per-party Bradley-Terry preference fits, seed candidates, and Nash search summaries. The path is off the rollback path and cannot affect deterministic gameplay. --- ## Library Summary The three historical libraries below carry the core inference, LLM, and vision surface. The training, evaluation, MLOps, safety, and Phase 176-179 capabilities in Sections 18-22 ship as additional `@nous/*` packages under `libs/nous/`. | Library | Package | Module Count | Primary Capabilities | | --------------- | -------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **nous-core** | `@nous/core` | 84 | ONNX inference runtime, hardware providers (CUDA/Metal/WebGPU/etc.), model registry, format handling, session management, quantization, graph optimization, tensor operations, embedding generation, similarity search (ANN/BM25/hybrid), reranking | | **nous-llm** | `@nous/llm` | 77 | LLM serving pipeline, chat completion, function calling, structured output, dynamic/continuous batching, KV cache, speculative decoding, multi-GPU parallelism, prompt template system, few-shot management, CoT/ToT/GoT reasoning, agent lifecycle, multi-agent coordination, document ingestion, chunking, vector indexing, RAG context retrieval | | **nous-vision** | `@nous/vision` | 40 | Object detection/segmentation, facial analysis, OCR, scene understanding, depth estimation, pose estimation, VQA, image captioning, Stable Diffusion (SD 1.5/2.x/XL/SD3), Flux, image-to-image, inpainting, outpainting, style transfer, ESRGAN upscaling | ### Domain Ownership Boundaries Nous owns model/runtime sovereignty and ML infrastructure. The following boundaries define where Nous ends and other domains begin: - **Iris** owns assistant UX and the end-user experience of privacy-mode LLM features; Nous provides the underlying local inference engine. - **Sophia** owns research knowledge management and synthesis workflows; Nous provides the LLM reasoning primitives. - **Isis** owns generative factory workflows and asset pipelines; Nous provides the model inference that powers individual generation steps. - **Gaia** owns weather and climate domain models; Nous provides the training and inference substrate those models run on. - **Concordia** coordinates cross-domain bargaining; Nous provides the preference-inference, agreement-search, and sealed-memory primitives that the bargaining engine is built from. - **Kuanyin and Themis** own safety and governance policy signals; Nous provides the safety classifiers and Model Card records that those policies evaluate. ## 23. ML Sovereignty Data and Model Programs (Phases 85–96) Nous is the center of the ML-sovereignty program; the phases below name the capability envelopes that Sections 18–22 implement or will implement: - **Data flywheel (Phases 85–86)** — passive training-signal capture from the domains: each participating domain ships a `training-data` library (`libs//training-data` exists today for Isis, Sophia, Hathor, Psyche, Veritas, Metis, Euterpe, Kuanyin, and Kalika) that emits consent-gated, anonymized signals in a shared flywheel envelope. Nous owns the envelope contract, ingestion, and downstream use. - **Dataset management and synthetic data (Phase 87)** — dataset registry, versioning, quality gates, deduplication, and synthetic-data generation (self-instruct, distillation corpora, simulation-derived data) feeding the training stack in Section 18. - **Domain-specific training pipelines (Phase 94)** — per-domain fine-tuning recipes that turn flywheel datasets into domain adapters (LoRA and full-parameter) with domain-owned evaluation sets; the owning domain defines quality, Nous runs the pipeline. - **Open-weight model integration (Phase 96)** — the evaluated open-weight model catalog (Llama-family and other current open-weight releases) with per-model licensing review, quantization profiles, and serving templates, kept current as the open-weight frontier moves. GPU training clusters (Phase 95) are provisioned by Shared infrastructure (RunPod serverless and reserved capacity) with Nous owning job scheduling and utilization; serving, edge, and MLOps envelopes are Sections 20 and 18–19. **Neith GPU compute-shader bridge (Phase 169).** Where Nous inference and tensor operations run on the sovereign engine's GPU compute framework, they bridge through `@neith/compute-ml` (the ML surface of the Phase 169 `compute-*` framework): Nous owns model/inference semantics, Neith owns the cross-backend GPU dispatch. Isis's self-hosted generation paths use the same bridge for GPU compute. ## 24. AI Platform Completion (Phase 98) Phase 98 rounds out the AI platform. Several of its scope areas map to existing sections — LLM observability (Sections 20, 22.2), A2A protocol and agent infrastructure (Section 15), unified RAG (Section 16), prompt engineering (Section 12), knowledge graph (Section 16 / Sophia), embedding infrastructure (Sections 8–9), advanced reasoning / test-time strategies (Section 13, ToT/GoT/ self-consistency), multimodal (Sections 17, 22.1), model compression and compiler-style optimization (Section 6), and EU AI Act / regulatory compliance and red-teaming (Section 21). The remaining envelopes are: - **Test-time compute scaling** — verifier-guided search, best-of-N with a reward/verifier model, and explicit reasoning-budget scaling on top of the Section 13 reasoning strategies. - **Semantic caching and cost efficiency** — embedding-keyed response caching so semantically equivalent requests reuse prior completions, with cost/quality tradeoff controls. - **Cross-domain agent memory** — a shared long-term agent memory substrate spanning domains (distinct from per-model context), consumed by Iris and domain agents. - **Simulation environments for agent training** — sandboxed task environments for training and evaluating agents (co-owned with the Phase 176 world-model and Phase 178 research tracks). - **Federated learning infrastructure** — privacy-preserving cross-tenant/ cross-domain training so models improve from distributed signals without centralizing raw data (the platform counterpart of the domain-specific federated learning noted for Kuanyin, Phase 32.18.5). - **ML supply-chain security** — model provenance, signing, and SBOM-style attestation for served and fine-tuned models. Co-owners: Iris (agent memory, assistant surfaces), Sophia (RAG, knowledge graph, evaluation), Psyche (multimodal, agent embodiment), and Shared (observability stack, compliance, supply-chain security).