# Nous Domain — Technical Specifications > Technical specifications for the Nous AI model infrastructure domain: library > inventory, type contracts, class APIs, configuration objects, and integration > points. Every type, field, and identifier in this document is traceable to > source under `libs/nous/`. This document is the authoritative type-level reference for the Nous domain. It complements the higher-level `features.md` (what Nous does) and `architecture.md` (how it is structured) by providing the exact TypeScript interfaces, enums, class signatures, and constant values that are exported from each `@nous/*` package. Engineers implementing a new Nous feature, writing a consumer in another domain, or reviewing a change for correctness should use this document alongside the actual source files under `libs/nous/`. The document is organised in dependency order: §1–§2 cover the package inventory and build system; §3–§9 cover `@nous/core` (the inference foundation); §10–§13 cover `@nous/llm`; §14–§17 cover the specialised packages (vision, audio, training, safety); §18–§19 cover generative media and Concordia primitives; and §20–§22 cover configuration, integration points, and acceptance criteria. --- ## 1. Library Inventory Nous is a **library-only domain**. It ships no `apps/nous` and no `services/nous` — the entire domain is a set of TypeScript workspace packages under `libs/nous/`. Twenty package directories exist; nineteen contain implemented source, and one (`platform`) is a config-only scaffold. The module counts below are non-spec `.ts` files in each `src/`; every implemented module in the six core packages has a sibling `*.spec.ts`. | Directory | Package | Source modules\* | Status | | ------------------------- | ------------------------------- | ---------------- | ----------- | | `core` | `@nous/core` | 84 | Implemented | | `llm` | `@nous/llm` | 78 | Implemented | | `vision` | `@nous/vision` | 103 | Implemented | | `audio` | `@nous/audio` | 81 | Implemented | | `training` | `@nous/training` | 137 | Implemented | | `safety` | `@nous/safety` | 64 | Implemented | | `image-control` | `@nous/image-control` | 50 | Implemented | | `video-control` | `@nous/video-control` | 43 | Implemented | | `video-removal` | `@nous/video-removal` | 48 | Implemented | | `video-to-audio` | `@nous/video-to-audio` | 21 | Implemented | | `advanced-speech` | `@nous/advanced-speech` | 31 | Implemented | | `generative-relighting` | `@nous/generative-relighting` | 23 | Implemented | | `portrait-animation` | `@nous/portrait-animation` | 23 | Implemented | | `joint-av-generation` | `@nous/joint-av-generation` | 34 | Implemented | | `agreement-search` | `@nous/agreement-search` | 22 | Implemented | | `preference-inference` | `@nous/preference-inference` | 24 | Implemented | | `cooperative-bargaining` | `@nous/cooperative-bargaining` | 2 | Implemented | | `concordia-sealed-memory` | `@nous/concordia-sealed-memory` | 5 | Implemented | | `platform` | `@nous/platform` | 0 | Scaffold | \* "Source modules" counts non-spec `.ts` files under `src/` (each module has a sibling `*.spec.ts` in `core`/`llm`/`vision`/`training`/`safety`/`audio`/the media packages; the four Concordia packages currently have no `.spec.ts` files). `@nous/platform` has `package.json`, `project.json`, three `tsconfig*.json`, and `vitest.config.ts`, but no `src/` directory. `tsconfig.base.json` declares a single wildcard path mapping: `"@nous/*": ["libs/nous/*/src/index.ts"]`. --- ## 2. Build System and Package Shape ### Source-Only Workspace Packages Every `libs/nous/*/package.json` sets `"main"` and `"types"` to `./src/index.ts` — packages are consumed directly as TypeScript source through the workspace; there is no published `dist/`. All packages declare `"type": "module"` and `"private": true`, version `0.1.0`. Package dependencies (`package.json`): - `@nous/core` — `optionalDependencies: { "onnxruntime-node": "^1.20.0" }`. - `@nous/safety` — `dependencies: { "@kalika/core", "@kalika/formal-verification" }` (both `workspace:*`). - `@nous/video-removal` — `dependencies: { "bullmq": "catalog:" }`. - `@nous/agreement-search`, `@nous/cooperative-bargaining`, `@nous/preference-inference` — each depends on `@concordia/contracts` (`workspace:*`) and `zod` (`catalog:`). - `@nous/concordia-sealed-memory` — depends on `zod` (`catalog:`). - All others — only `devDependencies` (`@types/node`, `typescript`, `vitest`, all `catalog:`). ### Nx Build Targets Each `project.json` (e.g. `libs/nous/core/project.json`) defines four targets that apply uniformly across all implemented packages. The `typecheck` target runs `tsc --noEmit` in isolation, which is important because Nx's duplicate-project detection can fail in worktree environments — in that case run the typecheck directly with `npx tsc --noEmit` from the package directory. | Target | Executor | Notes | | ----------- | --------------------- | ----------------------------------------------------------------- | | `build` | `@nx/esbuild:esbuild` | `format: ["esm"]`, `bundle: false`, output `libs/nous//dist` | | `lint` | `@nx/eslint:lint` | Lints `libs/nous//**/*.ts` | | `test` | `@nx/vite:test` | Config `libs/nous//vitest.config.ts` | | `typecheck` | `nx:run-commands` | `tsc --noEmit -p libs/nous//tsconfig.lib.json` | Project `tags` are `["scope:nous", "layer:infra", "type:lib"]`. Project `name`s are hyphenated (`nous-core`, `nous-llm`, …). --- ## 3. Core Type System (`@nous/core`) Source: `libs/nous/core/src/types.ts`. The barrel `libs/nous/core/src/index.ts` re-exports 83 modules plus `types.ts`. These base types are the currency of the entire `@nous/core` API — every execution provider, session, inference request, and batch result is built from them. Understanding `OnnxExecutionProviderName` (an open union that allows unknown providers), `OnnxTensorLike` (the lightweight tensor shape the engine accepts), and `ManagedOnnxSession` (the reference- counted session handle) is the prerequisite for reading any other section. ### ONNX Provider and Tensor Types ```typescript type OnnxExecutionProviderName = | 'cpu' | 'cuda' | 'rocm' | 'dml' | 'coreml' | 'metal' | 'webgpu' | 'vulkan' | 'xnnpack' | 'nnapi' | 'qnn' | 'npu' | (string & {}); // open union — unknown providers permitted type OnnxTensorType = | 'float32' | 'float64' | 'int8' | 'uint8' | 'int16' | 'uint16' | 'int32' | 'uint32' | 'int64' | 'uint64' | 'bool'; type OnnxTensorData = | Float32Array | Float64Array | Int8Array | Uint8Array | Int16Array | Uint16Array | Int32Array | Uint32Array | BigInt64Array | BigUint64Array; ``` ### Core Interfaces The following interfaces are the building blocks for every `@nous/core` operation. `OnnxSessionOptions` maps directly to ONNX Runtime session configuration; `NousOnnxSessionConfig` is the higher-level config passed to `OnnxSessionManager.acquire`; `ManagedOnnxSession` is the handle that callers receive and must `release()` when done. ```typescript interface OnnxExecutionProviderConfig { readonly name: OnnxExecutionProviderName; readonly options?: Readonly>; } interface OnnxTensorLike { readonly type: string; readonly data: TData; readonly dims: readonly number[]; readonly size?: number; } interface OnnxSessionOptions { readonly executionProviders?: readonly OnnxExecutionProviderConfig[]; readonly graphOptimizationLevel?: 'disabled' | 'basic' | 'extended' | 'all'; readonly executionMode?: 'sequential' | 'parallel'; readonly enableCpuMemArena?: boolean; readonly enableMemPattern?: boolean; readonly interOpNumThreads?: number; readonly intraOpNumThreads?: number; readonly extra?: Readonly>; readonly logSeverityLevel?: number; readonly logVerbosityLevel?: number; } type OnnxModelSource = string | ArrayBuffer | Uint8Array; interface OnnxInferenceSessionLike { readonly inputNames: readonly string[]; readonly outputNames: readonly string[]; run( feeds: Record, fetches?: readonly string[] ): Promise>; release?: () => void | Promise; dispose?: () => void | Promise; } interface OnnxRuntimeModule { readonly Tensor: new ( type: OnnxTensorType, data: OnnxTensorData, dims: readonly number[] ) => OnnxTensorLike; readonly InferenceSession: { create( source: OnnxModelSource, options?: OnnxSessionOptions | Record ): Promise; }; readonly env?: Record; } interface NousOnnxSessionConfig { readonly model: OnnxModelSource; readonly preferredProviders?: readonly OnnxExecutionProviderName[]; readonly sessionOptions?: Omit; readonly allowProviderFallback?: boolean; } interface ManagedOnnxSession { readonly key: string; readonly session: OnnxInferenceSessionLike; readonly providers: readonly OnnxExecutionProviderConfig[]; release(): Promise; } ``` ### Inference Request/Result Types `OnnxInferenceRequest` packages a session configuration, feed tensors, and optional fetch names into a single value that `OnnxInferenceEngine.run` consumes. `OnnxBatchInferenceRequest` adds a list of items and a mode: `independent` runs them concurrently with optional error tolerance, while `stacked` concatenates all items into one batch-dimension call. ```typescript interface TensorInput { readonly type?: OnnxTensorType; readonly data: ArrayLike; readonly dims: readonly number[]; } interface OnnxInferenceRequest { readonly session: NousOnnxSessionConfig; readonly feeds: Readonly>; readonly fetches?: readonly string[]; readonly validateFeeds?: boolean; readonly validateFetches?: boolean; } interface OnnxInferenceResult { readonly outputs: Readonly>; readonly sessionKey: string; readonly providers: readonly OnnxExecutionProviderConfig[]; } type OnnxBatchInferenceMode = 'independent' | 'stacked'; interface OnnxBatchInferenceRequest { readonly session: NousOnnxSessionConfig; readonly items: readonly OnnxBatchInferenceItem[]; readonly fetches?: readonly string[]; readonly validateFeeds?: boolean; readonly validateFetches?: boolean; readonly mode?: OnnxBatchInferenceMode; readonly maxConcurrency?: number; readonly continueOnError?: boolean; } interface OnnxBatchInferenceItemResult { readonly index: number; readonly outputs?: Readonly>; readonly error?: Error; } ``` ### Runtime Loader `loadOnnxRuntime` is what makes `onnxruntime-node` optional. It loads the runtime lazily via `createRequire` and validates the module shape before returning it. If the package is not installed, it throws `OnnxRuntimeNotAvailableError` — a typed error that callers can catch and handle gracefully (e.g., by reporting a clear "GPU inference unavailable" message rather than an unhandled `MODULE_NOT_FOUND`). Callers can also supply a custom loader via `OnnxRuntimeLoaderOptions.load` for testing or alternative runtimes. Source: `libs/nous/core/src/runtime-loader.ts`. ```typescript const DEFAULT_ONNX_RUNTIME_PACKAGE = 'onnxruntime-node'; interface OnnxRuntimeLoaderOptions { readonly runtimePackage?: string; readonly load?: (moduleSpecifier: string) => Promise; } class OnnxRuntimeNotAvailableError extends Error { readonly runtimePackage: string; } function loadOnnxRuntime( options?: OnnxRuntimeLoaderOptions ): Promise; ``` `loadOnnxRuntime` resolves `onnxruntime-node` (or a custom package) via `createRequire`, validates the module shape (`Tensor` constructor + `InferenceSession.create`), unwraps a `default` export if needed, and throws `OnnxRuntimeNotAvailableError` when the module is missing. The runtime is an **optional dependency**, so packages compile without it installed. --- ## 4. Execution Providers Each execution provider is a dedicated module that knows how to detect its own hardware, recommend a configuration, and produce `OnnxSessionOptions` with the correct `executionProviders` array. The shared `execution-providers.ts` module provides the registry and resolution logic that selects providers at session creation time. Source: `libs/nous/core/src/execution-providers.ts` plus per-provider modules (`cpu-execution-provider.ts`, `cuda-execution-provider.ts`, `rocm-execution-provider.ts`, `metal-execution-provider.ts`, `webgpu-execution-provider.ts`, `vulkan-execution-provider.ts`, `dml-execution-provider.ts`, `npu-execution-provider.ts`). ### Provider Registry and Resolution `KNOWN_EXECUTION_PROVIDERS` is the canonical list of twelve known backend names. `DEFAULT_PROVIDER_PREFERENCE` is the order in which they are tried when the caller does not specify a preference. `PROVIDER_ALIASES` maps informal spellings to canonical names so callers are not penalised for using `directml` instead of `dml`. `resolveExecutionProviders` is the core function — it walks the requested list, drops anything unavailable, and appends `cpu` if it is not already present. ```typescript const KNOWN_EXECUTION_PROVIDERS = [ 'cpu', 'cuda', 'rocm', 'dml', 'coreml', 'metal', 'webgpu', 'vulkan', 'xnnpack', 'nnapi', 'qnn', 'npu', ] as const; // 12 known providers const DEFAULT_PROVIDER_PREFERENCE = [ 'cuda', 'rocm', 'dml', 'coreml', 'metal', 'webgpu', 'vulkan', 'nnapi', 'qnn', 'xnnpack', 'cpu', ] as const; ``` `execution-providers.ts` also defines a `PROVIDER_ALIASES` map (e.g. `directml`/`dx12` → `dml`, `mps` → `metal`, `ane`/`neural_engine` → `npu`, `core-ml` → `coreml`). ```typescript interface ExecutionProviderEnvironment { readonly platform: NodeJS.Platform; readonly arch: NodeJS.Architecture; readonly cudaAvailable: boolean; readonly rocmAvailable: boolean; readonly dmlAvailable: boolean; readonly coremlAvailable: boolean; readonly metalAvailable: boolean; readonly webgpuAvailable: boolean; readonly vulkanAvailable: boolean; readonly nnapiAvailable: boolean; readonly qnnAvailable: boolean; readonly npuAvailable: boolean; } interface ExecutionProviderResolution { readonly requested: readonly OnnxExecutionProviderName[]; readonly resolved: readonly OnnxExecutionProviderName[]; readonly unavailable: readonly ExecutionProviderUnavailable[]; readonly fallbackApplied: boolean; readonly environment: ExecutionProviderEnvironment; } interface ExecutionProviderUnavailable { readonly provider: OnnxExecutionProviderName; readonly reason: string; // 'unknown_provider' | 'unavailable_in_environment' } ``` Exported functions: `detectExecutionProviderEnvironment`, `isExecutionProviderAvailable`, `getDefaultProviderPreference`, `normalizeExecutionProviderName`, `normalizeExecutionProviders`, `toExecutionProviderConfigs`, `resolveExecutionProviders`, `applyExecutionProvidersToSessionOptions`, `resolveSessionExecutionProviders`. `detectExecutionProviderEnvironment` reads the process environment for hardware signals (e.g. `CUDA_PATH`, `CUDA_HOME`, `ROCM_HOME`, `VULKAN_SDK`, `LD_LIBRARY_PATH` tokens) and `process.platform` to populate the environment struct. `resolveExecutionProviders` walks the requested list (or `DEFAULT_PROVIDER_PREFERENCE`), drops unavailable/unknown providers, and always appends a `cpu` fallback unless one is present. ### CPU Provider The CPU provider module detects the hardware profile (core count, platform, architecture) and translates an optimization goal — latency, throughput, or balanced — into specific thread counts for intra-op and inter-op parallelism. All other provider modules follow the same shape: a hardware-profile detector, a `recommend*Config` function, and a provider class that produces `OnnxSessionOptions`. Source: `cpu-execution-provider.ts`. ```typescript type CpuOptimizationGoal = 'latency' | 'throughput' | 'balanced'; type CpuExecutionMode = 'sequential' | 'parallel'; interface CpuHardwareProfile { readonly logicalCores: number; readonly estimatedPhysicalCores: number; readonly model: string; readonly platform: NodeJS.Platform; readonly arch: NodeJS.Architecture; } interface CpuExecutionProviderConfig { readonly optimizationGoal?: CpuOptimizationGoal; readonly useXnnpack?: boolean; readonly useArena?: boolean; readonly intraOpNumThreads?: number; readonly interOpNumThreads?: number; readonly executionMode?: CpuExecutionMode; readonly enableCpuMemArena?: boolean; readonly enableMemPattern?: boolean; } class CpuExecutionProvider { constructor(profile?: CpuHardwareProfile); getHardwareProfile(): CpuHardwareProfile; recommendConfig( goal?: CpuOptimizationGoal ): Required; createSessionOptions( config?: CpuExecutionProviderConfig, baseOptions?: Omit ): CpuSessionBuildResult; } ``` `detectCpuHardwareProfile` reads `node:os` `cpus()`; thread-count recommendations are derived from estimated physical cores and the optimization goal. The other provider modules follow the same shape — each exposes a hardware-profile detector, a `recommend*Config` function, and a provider class producing `OnnxSessionOptions` with `executionProviders` populated. --- ## 5. Inference Engine and Session Manager `OnnxSessionManager` and `OnnxInferenceEngine` are the two central classes for running inference. The session manager is the lower-level object: it owns the reference-counted cache of live ONNX sessions. The inference engine is the higher-level facade: it wraps the session manager and adds model loading, feed/fetch validation, and batch modes. Most consumers use the inference engine directly rather than the session manager. ### `OnnxSessionManager` Source: `libs/nous/core/src/session-manager.ts`. ```typescript interface OnnxSessionManagerOptions { readonly preferredProviders?: readonly OnnxExecutionProviderName[]; readonly allowProviderFallback?: boolean; // default true readonly maxCachedSessions?: number; // default 8 readonly providerEnvironment?: ExecutionProviderEnvironment; readonly providerOverrides?: ExecutionProviderOverrides; readonly strictKnownProviders?: boolean; // default false } interface OnnxSessionManagerStats { readonly totalSessions: number; readonly totalReferences: number; readonly sessions: readonly { key: string; providers: readonly OnnxExecutionProviderConfig[]; references: number; }[]; } class OnnxSessionManager { constructor(runtime: OnnxRuntimeModule, options?: OnnxSessionManagerOptions); acquire(config: NousOnnxSessionConfig): Promise; getStats(): OnnxSessionManagerStats; closeSession(key: string): Promise; disposeAll(): Promise; } ``` Behaviour: sessions are cached under a stable key derived from the model source (path string, or an FNV-style fingerprint of the bytes), the resolved provider list, the session options, and the fallback flag. `acquire` reference-counts entries; concurrent `acquire` calls for the same key share one in-flight creation. When the cache reaches `maxCachedSessions`, the least-recently-used zero-reference entry is evicted and its session released. If the primary provider fails and fallback is allowed, the manager retries with a `cpu`-only configuration. `closeSession` only succeeds when references are 0. ### `OnnxInferenceEngine` Source: `libs/nous/core/src/inference-engine.ts`. ```typescript interface OnnxInferenceEngineOptions { readonly runtime?: OnnxRuntimeModule; readonly loaderOptions?: OnnxRuntimeLoaderOptions; readonly preferredProviders?: readonly OnnxExecutionProviderName[]; readonly allowProviderFallback?: boolean; readonly maxCachedSessions?: number; readonly validateInputNames?: boolean; // default true readonly validateOutputNames?: boolean; // default true readonly modelLoader?: OnnxModelLoader; readonly modelLoaderOptions?: OnnxModelLoaderOptions; } class OnnxInferenceEngine { static create(options?: OnnxInferenceEngineOptions): Promise; getRuntime(): OnnxRuntimeModule; run(request: OnnxInferenceRequest): Promise; runBatch(request: OnnxBatchInferenceRequest): Promise; getSessionStats(): OnnxSessionManagerStats; registerModel(model: OnnxModelRegistration): void; unregisterModel(modelId: string): boolean; listRegisteredModels(): /* registered models */; preloadModels(modelIds: readonly string[]): Promise<...>; resolveModel(model: string): Promise; closeIdleSession(sessionKey: string): Promise; dispose(): Promise; } ``` `run` resolves the model through `OnnxModelLoader`, acquires a managed session, optionally validates feed/fetch names against `session.inputNames` / `session.outputNames` (throwing on unknown names), converts each feed to a runtime `Tensor`, and runs inference. `runBatch` supports two modes: `independent` (each item run concurrently up to `maxConcurrency`, with optional `continueOnError`) and `stacked` (items concatenated along the batch dimension into a single `run` call, then split back out — requires matching feed keys, tensor types, and non-batch dimensions, and rejects `continueOnError`). ### `OnnxModelLoader` Source: `libs/nous/core/src/model-loader.ts`. ```typescript interface OnnxModelRegistration { readonly id: string; readonly source: OnnxModelSource; readonly checksumSha256?: string; readonly metadata?: Readonly>; } interface ResolvedOnnxModel { readonly id?: string; readonly source: OnnxModelSource; readonly localPath?: string; readonly fromCache: boolean; readonly checksumSha256?: string; } interface OnnxModelLoaderOptions { readonly cacheDir?: string; // default /.cache/nous/models readonly fetch?: FetchLike; } type FetchLike = (url: string) => Promise; ``` The loader registers models by id, resolves string sources (treating HTTP URLs as downloadable, caching to `cacheDir`), verifies SHA-256 checksums via `node:crypto`, and exposes `preload`. Buffer/`ArrayBuffer` sources pass through in memory. --- ## 6. Model Management The model management layer sits between the application's model configuration and the inference engine's session creation. `ModelRegistry` is the central catalogue — it maps model IDs and aliases to metadata records. `OnnxModelLoader` handles the actual loading: string sources are treated as paths or downloadable URLs; `ArrayBuffer`/`Uint8Array` sources pass through in memory. The other model modules in `@nous/core` (versioning, download, checksum verification, conversion, benchmarking, etc.) each follow the same typed `*Options`/`*Result` pattern. ### `ModelRegistry` Source: `libs/nous/core/src/model-registry.ts`. ```typescript type ModelModality = 'text' | 'vision' | 'audio' | 'multimodal' | 'embedding'; interface ModelRegistryModel { readonly id: string; readonly displayName?: string; readonly provider?: string; readonly family?: string; readonly modality?: ModelModality; readonly task?: string; readonly source?: string; readonly tags?: readonly string[]; readonly capabilities?: readonly string[]; readonly metadata?: Readonly>; readonly createdAt?: number; readonly updatedAt?: number; readonly deprecated?: boolean; } interface ModelRegistryRegistration extends ModelRegistryModel { readonly aliases?: readonly string[]; } interface ModelRegistryListFilter { readonly provider?: string; readonly family?: string; readonly modality?: ModelModality; readonly task?: string; readonly tag?: string; readonly capability?: string; readonly includeDeprecated?: boolean; // default false } interface ModelRegistryStats { readonly totalModels: number; readonly deprecatedModels: number; readonly aliasCount: number; readonly defaultModelId?: string; } class ModelRegistry { registerModel( reg: ModelRegistryRegistration, options?: RegisterModelOptions ): void; unregisterModel(modelId: string): boolean; addAlias(modelId: string, alias: string): void; removeAlias(alias: string): boolean; resolveModelId(modelIdOrAlias: string): string | undefined; getModel(modelIdOrAlias: string): ModelRegistryModel | undefined; requireModel(modelIdOrAlias: string): ModelRegistryModel; setDefaultModel(modelIdOrAlias: string): void; getDefaultModel(): ModelRegistryModel | undefined; listModels(filter?: ModelRegistryListFilter): readonly ModelRegistryModel[]; searchModels( query: string, options?: ModelSearchOptions ): readonly ModelRegistryModel[]; listAliases(modelIdOrAlias: string): readonly string[]; getStats(): ModelRegistryStats; } ``` `RegisterModelOptions` is `{ overwrite?: boolean; setAsDefault?: boolean }`. Registration normalizes string fields (rejecting empties), deduplicates and sorts tags/capabilities, and validates aliases (an alias cannot equal a model id or collide with another model's id/alias). The first registered model becomes the default unless `setAsDefault` overrides it; `searchModels` substring-matches id, displayName, provider, family, task, tags, and capabilities. ### Other Model Modules Beyond the registry and loader, `@nous/core` ships a complete set of model lifecycle modules, each with its own typed `*Options`/`*Result` API: `model-versioning` (immutable version records), `model-metadata` (rich metadata storage), `model-discovery` (Hugging Face Hub and local directory scanning), `model-download` (HTTP download with progress tracking), `checksum-verification` (SHA-256 integrity checks), `model-conversion` (cross-format conversion), `format-detection` (header-byte-based format identification), `model-caching` (in-memory weight cache), `model-eviction` (LRU/LFU eviction policies), `model-preloading` (startup-time preloading), `warm-model-pool` (pre-initialized session pools), `model-benchmarking` (tokens/s, latency percentiles), and `model-profiling` (per-layer time and memory profiling). Format reader modules: `safetensors`, `gguf`, `ggml`, `pytorch-loading`, `tensorflow-loading`, `jax-loading`, `tensor`. --- ## 7. Inference Pipeline Primitives All inference pipeline primitives live flat in `libs/nous/core/src/`. The selected primitives below have their full types documented; the remaining modules follow the same typed-class pattern and are listed at the end of this section. ### Dynamic Batching `DynamicBatchScheduler` buffers individual inference requests and dispatches them as groups, improving GPU utilisation by reducing per-request kernel launch overhead. Requests are keyed by a batch key (default: a stable serialisation of session + fetches + flags) so requests for different models queue independently. Source: `dynamic-batching.ts`. ```typescript interface DynamicBatchExecutor { runBatch( request: OnnxBatchInferenceRequest ): Promise; } interface DynamicBatchingOptions { readonly maxBatchSize?: number; // default 8 readonly maxQueueDelayMs?: number; // default 8 readonly mode?: OnnxBatchInferenceMode; readonly maxConcurrency?: number; readonly maxQueueSize?: number; readonly batchKeyFn?: (request: OnnxInferenceRequest) => string; } interface DynamicBatchingStats { readonly pendingRequests: number; readonly batchesExecuted: number; readonly requestsProcessed: number; readonly requestsFailed: number; readonly requestsDropped: number; readonly averageBatchSize: number; readonly maxObservedBatchSize: number; } class DynamicBatchScheduler { /* submit, stats, … */ } ``` Requests are grouped per batch key (default key = stable serialization of session + fetches + validation flags), flushed when `maxBatchSize` is reached or `maxQueueDelayMs` elapses. ### KV Cache `KvCacheManager` explicitly manages the key-value buffers that LLM generation accumulates across forward passes. Memory is reserved up front (`maxTotalBytesReserved`), with per-layer `Float32Array` buffers allocated at `allocate()` time. The byte cost formula is `capacityTokens × numHeads × headSize × 4 (bytes/float) × 2 (K+V)` per layer, summed across all layers — use this to size the cache budget for a given model. Source: `kv-cache.ts`. ```typescript interface KvCacheConfig { readonly modelId: string; readonly layerCount: number; readonly numHeads: number; readonly headSize: number; readonly initialCapacityTokens?: number; } interface KvLayerAppendInput { readonly layer: number; readonly tokenCount: number; readonly keys: Float32Array; readonly values: Float32Array; } interface KvCacheSnapshot { readonly cacheId: string; readonly modelId: string; readonly tokenCount: number; readonly layers: readonly KvLayerSnapshot[]; } interface KvCacheStats { readonly cacheCount: number; readonly totalBytesReserved: number; readonly evictions: number; readonly cacheIds: readonly string[]; } interface KvCacheManagerOptions { readonly maxTotalBytesReserved?: number; // default 512 MiB } class KvCacheManager { /* allocate / append / read / evict / stats */ } ``` Per-layer key/value buffers are `Float32Array`s; reserved bytes are computed as `capacityTokens * numHeads * headSize * 4 * 2` summed across layers. ### Speculative Decoding `SpeculativeDecoder` coordinates a fast draft model and a slower target model. The draft model proposes a batch of candidate tokens; the target model verifies them in a single forward pass. Accepted tokens reduce the number of full-model passes, cutting wall-clock generation latency by 2–3x. `createGreedyTargetModel` provides a pre-built target verifier for greedy decoding. Source: `speculative-decoding.ts`. ```typescript interface SpeculativeDraftModel { /* draft token proposal */ } interface SpeculativeTargetVerification { /* per-token accept/reject */ } interface SpeculativeTargetModel { /* full-model verification */ } interface SpeculativeDecodingOptions { /* numSpeculativeTokens etc. */ } interface SpeculativeDecodingRequest { … } interface SpeculativeDecodingResult { … } function createGreedyTargetModel(...): SpeculativeTargetModel; class SpeculativeDecoder { … } ``` ### Multi-LoRA Serving `MultiLoraServingManager` allows multiple LoRA adapters to be registered against a single base model, then selected per-request by adapter ID. Adapter weights are a small fraction of the base model's size (roughly 1–5%), so serving ten adapters from one base model requires far less memory than loading ten separate models. Source: `multi-lora-serving.ts`. ```typescript interface LoraAdapterRegistration { /* adapter id, base model, … */ } interface MultiLoraServingOptions { … } interface MultiLoraRequest { … } interface MultiLoraExecutionGroup { … } interface MultiLoraExecutionPlan { … } interface MultiLoraServingStats { … } class MultiLoraServingManager { … } ``` ### Additional Pipeline Modules The following modules are implemented with their own `*.spec.ts`, each following the same typed-class pattern as the modules documented above. They cover the remaining throughput, attention, parallelism, caching, generation control, and graph optimization capabilities described in `features.md` §3–§6: - **Throughput / serving**: `continuous-batching`, `llm-serving` - **Attention and memory**: `paged-attention`, `flash-attention` - **Multi-GPU parallelism**: `tensor-parallelism`, `pipeline-parallelism`, `model-sharding`, `moe-parallelism` - **Caching**: `prefix-caching`, `radix-attention`, `prompt-caching` - **Adapter management**: `adapter-switching`, `context-extension` - **Generation control**: `streaming-generation`, `stop-sequences`, `logit-processors`, `temperature-sampling`, `top-k-top-p-sampling`, `beam-search` - **Graph optimization**: `graph-optimization`, `memory-planning`, `operator-fusion` --- ## 8. Quantization `quantization.ts` provides the low-level numerical conversion functions that underpin the quantization pipeline. `computeSymmetricQuantParams` and `computeAsymmetricQuantParams` derive scale and zero-point from a calibration range. The `float32/float16` conversion pairs handle FP16 round-trips. `quantizeInt8`/`dequantizeInt8` handle the INT8 path; `packInt4`/`unpackInt4` pack two nibbles per byte for INT4. These functions are called by the higher-level `model-optimization-pipeline` module but can also be used directly for custom quantization workflows. Source: `libs/nous/core/src/quantization.ts`. ```typescript type QuantizationPrecision = 'fp16' | 'int8' | 'int4'; type QuantizationMethod = 'dynamic' | 'static'; interface QuantizationConfig { /* precision + method */ } interface QuantizationParams { … } interface CalibrationRange { … } type QuantizedTensor = | QuantizedTensorFp16 | QuantizedTensorInt8 | QuantizedTensorInt4; interface QuantizationBatchResult { … } ``` Exported functions include `computeSymmetricQuantParams`, `computeAsymmetricQuantParams`, `float32ToFloat16`, `float16ToFloat32`, `convertFloat32ToFloat16`, `convertFloat16ToFloat32`, `quantizeInt8`, `dequantizeInt8`, `packInt4`, `unpackInt4`. INT4 values are nibble-packed two per byte. --- ## 9. Embeddings and Similarity Search The embedding and search modules in `@nous/core` form the vector search stack that underpins RAG, semantic search, and recommendation. `EmbeddingGenerator` is the entry point: it manages a priority-ordered provider list and dispatches requests by modality. `SimilaritySearchIndex` is the retrieval half: it stores `(id, vector, metadata)` triples and supports cosine, dot, and Euclidean queries. Companion modules add ANN indexing, BM25 hybrid search, and cross-encoder reranking on top of this foundation. ### Embedding Generation Source: `libs/nous/core/src/embedding-generation.ts`. ```typescript type EmbeddingModality = 'text' | 'image' | 'audio' | 'multimodal'; type EmbeddingContent = | string | Uint8Array | ArrayBuffer | readonly (string | Uint8Array | ArrayBuffer)[]; interface EmbeddingInput { readonly modality: EmbeddingModality; readonly content: EmbeddingContent; readonly metadata?: Readonly>; } interface EmbeddingGenerationRequest { readonly modelId: string; readonly input: EmbeddingInput; readonly providerId?: string; readonly outputDimensions?: number; readonly timeoutMs?: number; readonly allowProviderFallback?: boolean; readonly metadata?: Readonly>; } interface EmbeddingProvider { readonly id: string; readonly supportedModalities: readonly EmbeddingModality[]; readonly supportedModels?: readonly string[]; readonly priority?: number; generate(request: EmbeddingProviderRequest): Promise; } interface EmbeddingGenerationResult { readonly providerId: string; readonly modelId: string; readonly modality: EmbeddingModality; readonly vector: readonly number[]; readonly dimensions: number; readonly createdAt: number; readonly metadata?: Readonly>; } interface EmbeddingGeneratorStats { readonly totalRequests: number; readonly successfulRequests: number; readonly failedRequests: number; readonly generatedVectors: number; readonly averageLatencyMs: number; readonly providers: readonly EmbeddingProviderStats[]; } class EmbeddingGenerator { /* registerProvider / generate / getStats */ } ``` Companion modules add modality-specific input processing and post-processing capabilities: `text-embeddings`, `image-embeddings`, `audio-embeddings`, `multi-modal-embeddings` handle input encoding for each modality; `embedding-normalization` L2-normalises vectors; `dimensionality-reduction` applies PCA or random projection; `batch-embedding` and `streaming-embedding` handle bulk processing; `embedding-caching` deduplicates on input hash; `embedding-compression`, `binary-embeddings`, and `matryoshka-embeddings` reduce storage footprint. ### Similarity Search Source: `libs/nous/core/src/similarity-search.ts`. ```typescript type SimilarityMetricName = 'cosine' | 'dot' | 'euclidean'; interface SimilaritySearchItem { readonly id: string; readonly embedding: readonly number[]; readonly metadata?: Readonly>; } interface SimilaritySearchQuery { readonly embedding: readonly number[]; readonly topK?: number; readonly minScore?: number; readonly filter?: Readonly>; readonly includeEmbeddings?: boolean; } interface SimilaritySearchResultItem { readonly id: string; readonly score: number; readonly rank: number; readonly metadata?: Readonly>; readonly embedding?: readonly number[]; } interface SimilaritySearchResult { readonly metric: SimilarityMetricName; readonly totalCandidates: number; readonly evaluatedCandidates: number; readonly durationMs: number; readonly items: readonly SimilaritySearchResultItem[]; } class SimilaritySearchIndex { /* index / search / stats */ } ``` Related modules extend `SimilaritySearchIndex` for different scale and accuracy requirements: `approximate-nearest-neighbor` (HNSW/IVF for billion-scale datasets), `exact-nearest-neighbor` (brute-force for small corpora where accuracy is paramount), `distance-metrics` (cosine, Euclidean, dot, Manhattan), `hybrid-search` (vector + BM25 with Reciprocal Rank Fusion), `reranking` (score-based re-ordering), `cross-encoder-reranking` (BERT cross-encoder two-stage pipeline). --- ## 10. LLM Completion APIs (`@nous/llm`) `@nous/llm` builds the higher-level language-model surface on top of `@nous/core`. Its two completion classes — `CompletionAPI` (text) and `ChatCompletionAPI` (conversation) — follow a _provider registry_ pattern: the application registers one or more provider implementations, and the API dispatches to them by ID with an optional fallback chain. This design decouples the API surface from any specific backend: an application can register an ONNX-backed local provider, a remote API provider, or a mock for testing, and the calling code stays unchanged. The barrel `libs/nous/llm/src/index.ts` re-exports 78 modules. ### Text Completion `CompletionAPI` is the lower-level interface for open-ended text generation. `CompletionAPIRequest` extends `CompletionRequest` with a `providerId` and optional `fallbackProviderIds`. The response records which provider was used, the model name reported by that provider, latency, and token usage. Source: `completion-api.ts`. ```typescript interface CompletionChoice { readonly index: number; readonly text: string; readonly finishReason?: 'stop' | 'length' | 'content_filter' | 'error'; } interface CompletionUsage { readonly promptTokens: number; readonly completionTokens: number; readonly totalTokens: number; } interface CompletionRequest { readonly prompt: string; readonly model?: string; readonly maxTokens?: number; readonly temperature?: number; readonly topP?: number; readonly stop?: readonly string[]; readonly metadata?: Readonly>; } interface CompletionProvider { readonly id: string; readonly models?: readonly string[]; complete(request: CompletionRequest): Promise; } interface CompletionAPIRequest extends CompletionRequest { readonly providerId?: string; readonly fallbackProviderIds?: readonly string[]; } interface CompletionResponse { readonly id: string; readonly providerId: string; readonly model: string; readonly createdAt: number; readonly latencyMs: number; readonly choices: readonly CompletionChoice[]; readonly usage: CompletionUsage; } class CompletionAPI { constructor(options?: CompletionAPIOptions); registerProvider(provider: CompletionProvider): void; unregisterProvider(providerId: string): boolean; setDefaultProvider(providerId: string): void; listProviders(): readonly string[]; complete(request: CompletionAPIRequest): Promise; completeText(request: CompletionAPIRequest): Promise; getStats(): CompletionAPIStats; } ``` The completion layer is **provider-registry based**: callers register `CompletionProvider` implementations, then dispatch by `providerId` (with optional `fallbackProviderIds`). `CompletionAPIOptions` is `{ now?: () => number; idFactory?: () => string }`. ### Chat Completion `ChatCompletionAPI` is the conversational interface. It uses the same provider-registry pattern as `CompletionAPI`. The `tool_calls` finish reason in `ChatCompletionChoice` signals that the model wants to invoke a tool rather than producing a final response — the caller is responsible for handling the tool call and continuing the conversation. Source: `chat-completion-api.ts`. ```typescript type ChatMessageRole = 'system' | 'user' | 'assistant' | 'tool'; interface ChatMessage { readonly role: ChatMessageRole; readonly content: string; readonly name?: string; } interface ChatCompletionChoice { readonly index: number; readonly message: ChatMessage; readonly finishReason?: | 'stop' | 'length' | 'content_filter' | 'error' | 'tool_calls'; } interface ChatCompletionRequest { readonly messages: readonly ChatMessage[]; readonly model?: string; readonly maxTokens?: number; readonly temperature?: number; readonly topP?: number; readonly stop?: readonly string[]; readonly metadata?: Readonly>; } interface ChatCompletionProvider { readonly id: string; readonly models?: readonly string[]; complete( request: ChatCompletionRequest ): Promise; } ``` `ChatCompletionAPI` mirrors `CompletionAPI` (provider registry + fallback). ### Output Shaping and NLP The following modules cover structured output enforcement, NLP post-processing, and context management. Each is a dedicated source file with its own `*.spec.ts` under `libs/nous/llm/src/`: - **Completion modes**: `instruction-following`, `multi-turn-conversation`, `system-prompts`, `function-calling`, `tool-use-framework` - **Structured output**: `json-mode`, `structured-output`, `grammar-constraints`, `regex-constraints`, `schema-validation` - **Response processing**: `response-formatting`, `markdown-rendering`, `code-extraction`, `citation-extraction`, `fact-extraction` - **NLP**: `entity-extraction`, `sentiment-analysis`, `classification` ### Reasoning and Persona Reasoning and persona modules add layers of structured thinking and identity management on top of the core completion interfaces: - **Reasoning strategies**: `chain-of-thought`, `tree-of-thought`, `graph-of-thought`, `self-consistency`, `reflection-prompts`, `critique-prompts` - **Persona and context**: `persona-management`, `context-injection`, `memory-injection`, `retrieval-injection` --- ## 11. Prompt Templates The prompt template system provides a versioned, testable alternative to string concatenation for prompt construction. `PromptTemplateDefinition` is the input shape when registering a template; `PromptTemplateRecord` is the stored shape that adds a `revision` counter, timestamps, and normalised field defaults. `PromptTemplateRenderRequest` controls how the template is expanded: `strict` mode raises an error on missing variables; `missingValue` provides a fallback string; `stringifyObjects` serialises complex values to JSON. The `unresolvedVariables` field in the result identifies any variable that was not provided, even in non-strict mode. Source: `libs/nous/llm/src/prompt-template-system.ts`. ```typescript type PromptTemplateRole = 'system' | 'user' | 'assistant'; interface PromptTemplateMessage { readonly role: PromptTemplateRole; readonly content: string; } interface PromptTemplateDefinition { readonly id: string; readonly name: string; readonly description?: string; readonly version?: string; readonly messages: readonly PromptTemplateMessage[]; readonly variables?: readonly string[]; readonly tags?: readonly string[]; readonly metadata?: Readonly>; } interface PromptTemplateRecord { readonly id: string; readonly name: string; readonly description?: string; readonly version: string; readonly messages: readonly PromptTemplateMessage[]; readonly variables: readonly string[]; readonly tags: readonly string[]; readonly metadata: Readonly>; readonly revision: number; readonly createdAt: number; readonly updatedAt: number; } interface PromptTemplateRenderRequest { readonly templateId: string; readonly values?: Readonly>; readonly includeMetadata?: boolean; readonly strict?: boolean; readonly missingValue?: string; readonly stringifyObjects?: boolean; } interface PromptTemplateRenderResult { readonly templateId: string; readonly version: string; readonly messages: readonly PromptTemplateMessage[]; readonly unresolvedVariables: readonly string[]; readonly metadata?: Readonly>; } ``` Each registered template carries a `revision` counter that increments on update. `strict` render mode surfaces missing variables; `missingValue` supplies a fallback string; `stringifyObjects` serializes object values. Companion modules extend the template system with lifecycle management and example-based prompting: `prompt-versioning` (version pinning for reproducible deployments), `prompt-testing` (regression detection across template changes), `prompt-optimization` (systematic variation search), `few-shot-management` (input-output example collections), `example-selection` (embedding-similarity retrieval of the most relevant examples), `dynamic-examples` (context-adaptive example generation). --- ## 12. Agent Architecture `AgentArchitecture` is a runtime that manages the lifecycle of named agents and their task queues. An agent is defined by its `AgentDefinition` — a set of capabilities, concurrency limits, and pluggable module implementations (planner, goal decomposer, executor). Once registered, an agent transitions through lifecycle states (`stopped → idle → running → paused | error`) and processes tasks submitted via `submitTask`. Every state change produces a typed `AgentRuntimeEvent` that is recorded in the event log, making the full run history queryable for debugging and monitoring. Source: `libs/nous/llm/src/agent-architecture.ts`. ### State and Capability Enums The three enum types below define the vocabulary for agent and task state machines. `AgentCapability` is used as a tag on `AgentDefinition` to declare what the agent can do; consumers can query the runtime for agents with specific capabilities. ```typescript type AgentLifecycleState = 'stopped' | 'idle' | 'running' | 'paused' | 'error'; type AgentTaskState = | 'queued' | 'running' | 'completed' | 'failed' | 'cancelled' | 'timeout'; type AgentCapability = | 'planning' | 'goal-decomposition' | 'tool-use' | 'memory' | 'retrieval' | 'communication'; ``` ### Agent and Task Types `AgentDefinition` is the registration input; `AgentRecord` is the stored record that adds runtime state and task counters. `AgentModuleSet` must always include an `executor` (the module that actually runs a task step); the `planner` and `goalDecomposer` are optional modules for planning and decomposition respectively. The pluggable modules receive an `AgentPlanningContext` or `AgentExecutionContext` that provides state access and an event emitter — they do not call the runtime directly. ```typescript interface AgentTaskInput { readonly goal: string; readonly payload?: unknown; readonly context?: Readonly>; } interface AgentPlan { readonly summary: string; readonly steps: readonly string[]; readonly confidence?: number; } interface AgentSubtask { readonly id: string; readonly description: string; readonly priority?: number; readonly metadata?: Readonly>; } interface AgentDefinition { readonly id: string; readonly name: string; readonly description?: string; readonly capabilities?: readonly AgentCapability[]; readonly maxConcurrency?: number; readonly defaultTimeoutMs?: number; readonly modules: AgentModuleSet; readonly metadata?: Readonly>; } interface AgentModuleSet { readonly planner?: AgentPlannerModule; readonly goalDecomposer?: AgentGoalDecomposerModule; readonly executor: AgentExecutorModule; // required } interface AgentRecord { readonly id: string; readonly name: string; readonly description?: string; readonly capabilities: readonly AgentCapability[]; readonly state: AgentLifecycleState; readonly maxConcurrency: number; readonly defaultTimeoutMs: number; readonly metadata: Readonly>; readonly createdAt: number; readonly updatedAt: number; readonly activeTasks: number; readonly queuedTasks: number; readonly completedTasks: number; readonly failedTasks: number; } ``` The pluggable modules implement `plan(input, context)`, `decompose(input, context)`, and `execute(input, context)`. `AgentPlanningContext` extends a `getState`/`setState` accessor and exposes `agentId`, `taskId`, `now`, and an `emit(event, payload)` callback; `AgentExecutionContext` adds the resolved `plan` and `subtasks`. ### `AgentArchitecture` Runtime ```typescript class AgentArchitecture { constructor(options?: AgentArchitectureOptions); registerAgent(definition: AgentDefinition): AgentRecord; unregisterAgent(agentId: string): boolean; startAgent(agentId: string): AgentRecord; stopAgent(agentId: string, options?: { cancelQueued?: boolean }): AgentRecord; pauseAgent(agentId: string): AgentRecord; resumeAgent(agentId: string): AgentRecord; submitTask(request: AgentTaskRequest): string; // returns taskId runNext(agentId?: string): Promise; runAll(...): Promise<...>; runTask(taskId: string): Promise; cancelTask(taskId: string): boolean; getAgent(agentId: string): AgentRecord | undefined; listAgents(): readonly AgentRecord[]; getTask(taskId: string): AgentTaskRecord | undefined; listTasks(filter?: AgentTaskFilter): readonly AgentTaskRecord[]; getEvents(limit?: number): readonly AgentRuntimeEvent[]; getStats(): AgentArchitectureStats; } ``` The runtime records typed `AgentRuntimeEvent`s into an event log queryable via `getEvents`. The companion modules implement the pluggable capabilities declared in `AgentCapability`: - **Execution**: `planning-module`, `goal-decomposition`, `task-execution`, `action-selection`, `tool-selection`, `multi-step-reasoning`, `backtracking`, `error-recovery` - **Memory**: `agent-memory`, `short-term-memory`, `long-term-memory`, `episodic-memory`, `semantic-memory`, `working-memory` - **Multi-agent**: `agent-communication`, `multi-agent-systems`, `agent-coordination`, `agent-delegation`, `agent-supervision` --- ## 13. Document Ingestion and RAG `DocumentIngestion` is the entry point for turning raw files into indexed, searchable document chunks. It operates through a content-type-matched parser registry: the application registers parsers for `application/pdf`, `text/html`, `text/markdown`, etc., and the `ingest()` call dispatches to the matching parser, then runs chunking, embedding, and indexing. The result reports a status per document — `ingested`, `duplicate`, or `failed` — so batch ingestion can continue past individual failures. Source: `libs/nous/llm/src/document-ingestion.ts`. ```typescript type DocumentIngestionSourceType = 'text' | 'file' | 'url' | 'binary'; type DocumentIngestionResultStatus = 'ingested' | 'duplicate' | 'failed'; interface DocumentIngestionInput { /* source type + payload + metadata */ } type DocumentIngestionParser = (request: DocumentIngestionParseRequest) => DocumentIngestionParseResult | Promise; type DocumentIngestionParserMatcher = string | RegExp | ((contentType: string) => boolean); interface IngestedDocumentRecord { … } interface DocumentIngestionResult { … } interface DocumentIngestionBatchResult { … } interface DocumentIngestionStats { … } interface DocumentIngestionEvent { … } class DocumentIngestion { /* parser registry + ingest + stats + events */ } ``` `DocumentIngestion` registers content-type-matched parsers and reports a per-document status (`ingested` / `duplicate` / `failed`). The companion modules cover the full pipeline from raw bytes to searchable index: - **Chunking**: `chunking-strategies`, `semantic-chunking`, `sentence-chunking`, `token-chunking`, `recursive-chunking` - **Parsing**: `document-parsing`, `pdf-parsing`, `html-parsing`, `markdown-parsing`, `code-parsing` - **Extraction**: `table-extraction`, `image-extraction`, `metadata-extraction`, `link-extraction` - **Indexing and retrieval**: `vector-indexing`, `hybrid-indexing`, `incremental-indexing`, `index-optimization`, `context-retrieval` --- ## 14. Vision (`@nous/vision`) `@nous/vision` is the largest package in the domain by module count (103 modules). It covers both understanding (detection, recognition, OCR, scene analysis) and generation (Stable Diffusion, Flux, video synthesis). All models run locally through `@nous/core`'s session manager — there are no external vision API calls. The barrel `libs/nous/vision/src/index.ts` re-exports all 103 modules, covering still-image **and** video understanding/generation. ### Object Detection `ObjectDetection` returns `DetectedObject` records with pixel-space bounding boxes, normalised (0..1) bounding boxes, class labels from a COCO-style category union, and confidence scores. The `ObjectDetectionBatchResult` variant processes multiple images in one call. Source: `object-detection.ts`. ```typescript type ObjectCategory = /* fixed COCO-style category union */; interface BoundingBox { readonly x: number; readonly y: number; readonly width: number; readonly height: number; } interface NormalizedBoundingBox { … } // 0..1 coordinates interface DetectedObject { /* category, confidence, bounding boxes */ } interface ObjectDetectionRequest { … } interface ObjectDetectionResult { … } interface ObjectDetectionBatchResult { … } interface ObjectDetectionStats { … } class ObjectDetection { … } ``` ### Stable Diffusion `StableDiffusion` covers the SD 1.x and 2.x model families. `StableDiffusionModelId` is a closed union of the supported checkpoint identifiers. The `quality_preset` setting (`draft`, `balanced`, `quality`) trades step count against generation time. The `safety_policy` field controls whether NSFW-classified outputs are passed through (`none`), flagged (`warn`), or blocked (`block`). Source: `stable-diffusion.ts`. ```typescript type StableDiffusionModelId = 'sd-1.4' | 'sd-1.5' | 'sd-2.0' | 'sd-2.1' | 'sd-2.1-768'; type StableDiffusionQualityPreset = 'draft' | 'balanced' | 'quality'; type StableDiffusionSafetyPolicy = 'none' | 'warn' | 'block'; interface StableDiffusionGenerationRequest { … } interface StableDiffusionSafetyAssessment { … } interface StableDiffusionGeneratedImage { … } interface StableDiffusionGenerationResult { … } interface StableDiffusionBatchResult { … } interface StableDiffusionStats { … } class StableDiffusion { … } ``` `@nous/vision` ships dedicated modules for `sdxl-support`, `sd3-support`, `flux-support`, and a generic `diffusion-model-framework` / `video-diffusion-framework`. ### Module Groups The 103 modules in `@nous/vision` are organised into the following functional groups. Each module has its own `*.spec.ts` under `libs/nous/vision/src/`. - **Detection / classification**: `object-detection`, `image-classification`, `instance-segmentation`, `semantic-segmentation`, `panoptic-segmentation`, `anomaly-detection`, `multi-object-tracking`, `object-tracking`, `re-identification`, `crowd-analysis`, `traffic-analysis`. - **Facial**: `facial-recognition`, `face-landmark-detection`, `face-restoration`, `facial-reenactment`, `emotion-recognition`, `age-gender-estimation`. - **OCR / documents**: `ocr-engine`, `handwriting-recognition`, `document-layout`, `table-recognition`, `barcode-qr-recognition`, `license-plate-recognition`. - **Scene understanding**: `depth-estimation`, `surface-normal-estimation`, `pose-estimation`, `scene-detection`, `image-captioning`, `visual-question-answering`, `action-recognition`, `trajectory-prediction`. - **Image generation / editing**: `stable-diffusion`, `sdxl-support`, `sd3-support`, `flux-support`, `controlnet-integration`, `ip-adapter-support`, `image-to-image`, `inpainting`, `outpainting`, `background-removal`, `style-transfer`, `image-composition`, `image-editing`, `upscaling-esrgan`, `variation-generation`, `progressive-generation`, `consistent-character-generation`, `regional-prompting`, `pose-guided-generation`, `depth-aware-generation`, `edge-guided-generation`, `semantic-guided-generation`, `reference-based-generation`, `lora-management`, `dreambooth-training`, `textual-inversion`. - **Video**: `text-to-video`, `image-to-video`, `video-to-video`, `video-editing`, `video-inpainting`, `video-outpainting`, `video-upscaling`, `video-stylization`, `video-composition`, `video-classification`, `video-captioning`, `dense-video-captioning`, `video-question-answering`, `video-indexing`, `video-summarization`, `video-text-matching`, `frame-interpolation`, `slow-motion-generation`, `temporal-super-resolution`, `temporal-consistency`, `temporal-segmentation`, `temporal-grounding`, `moment-retrieval`, `shot-boundary-detection`, `scene-coherence-support`, `motion-transfer`, `rotoscoping-automation`. (Other modules — `turbo-lightning-modes`, `sparse-dit-support`, `moe-routing-support`, `llama-text-encoder-support`, `text-rendering-support`, `prompt-adherence-support`, `hidream-benchmark-parity`, `render-mode-support`, `quality-speed-tradeoff-controls` — are present in the barrel and implemented.) --- ## 15. Audio (`@nous/audio`) `@nous/audio` provides the speech, music, and audio analysis surface of Nous. All 81 modules run locally through the same ONNX session infrastructure as `@nous/core`. The package covers four broad areas: ASR (automatic speech recognition), TTS (text-to-speech synthesis and voice cloning), music generation, and audio analysis / restoration. Each module has its own `*.spec.ts`. Source: `libs/nous/audio/src/`, 81 modules. The 81 modules group into the four functional areas described below. - **Speech recognition**: `asr-engine`, `multi-language-asr`, `language-detection`, `custom-vocabulary`, `hotword-detection`, `confidence-scores`, `capitalization`, `code-switching-support`. - **Speech synthesis**: `neural-tts`, `multi-speaker-tts`, `few-shot-cloning`, `emotion-control`, `emphasis-control`, `accent-adaptation`, `language-mixing`. - **Music**: `music-generation-framework`, `melody-generation`, `harmony-generation`, `accompaniment-generation`, `chord-progression`, `chord-recognition`, `music-continuation`, `music-style-transfer`, `music-transcription`, `midi-generation`, `audiobook-generation`, `beat-detection`, `key-detection`. - **Analysis / restoration**: `audio-classification`, `audio-scene-classification`, `audio-segmentation`, `audio-tagging`, `genre-classification`, `mood-classification`, `audio-search`, `audio-fingerprinting`, `content-identification`, `audio-forensics`, `audio-enhancement`, `audio-restoration`, `audio-inpainting`, `audio-super-resolution`, `bandwidth-extension`, `declipping`, `denoising`, `noise-reduction`, `dereverberation`, `echo-cancellation`, `domain-adaptation`. --- ## 16. Training (`@nous/training`) `@nous/training` is the largest package in the domain (137 modules). It provides a full model-training surface — not only fine-tuning but also distributed training infrastructure, all major optimiser variants, alignment/preference tuning, diffusion model training, and a complete data pipeline. Two concrete V2 contracts are also implemented here: the anti-cheat classifier training loop (§16.2) and the preference alignment stack. Source: `libs/nous/training/src/`, 137 modules. ### Module Groups The 137 modules cover seven functional areas: - **Distributed training**: `distributed-training-system`, `distributed-data-loading`, `data-parallelism`, `model-parallelism`, `tensor-parallelism`, `pipeline-parallelism`, `zero-optimization`, `mixed-precision-training`, `gradient-checkpointing`, `gradient-accumulation`, `fault-tolerance`, `checkpoint-management`. - **Optimizers / schedules**: `learning-rate-schedulers`, `warmup-strategies`, `optimizer-selection`, `adamw-implementation`, `lamb-optimizer`, `adafactor-optimizer`, `eight-bit-optimizers`, `loss-functions`, `metric-tracking`. - **Fine-tuning**: `supervised-fine-tuning`, `instruction-tuning`, `chat-fine-tuning`, `lora-fine-tuning`, `qlora-fine-tuning`, `adapter-tuning`, `prefix-tuning`, `prompt-tuning`, `p-tuning`, `ia3-tuning`, `dora-tuning`, `peft-integration`, `multi-task-fine-tuning`. - **Continual learning / distillation**: `continual-learning`, `catastrophic-forgetting-prevention`, `knowledge-distillation`, `teacher-student-training`, `self-distillation`, `progressive-training`, `curriculum-learning`. - **Preference / alignment tuning**: `reward-modeling`, `preference-learning`, `comparison-data-collection`, `bradley-terry-model`, `ppo-training`, `dpo-training`, `ipo-training`, `kto-training`, `orpo-training`, `rlaif`, `rejection-sampling`, `best-of-n-sampling`, `iterative-refinement`, `self-play-training`, `constitutional-ai`, `red-teaming`, `safety-training`, `helpfulness-training`, `harmlessness-training`, `honesty-training`, `preference-aligned-inference`, `alignment-quality-metrics`. - **Diffusion training / acceleration**: `diffusion-dpo-training`, `diffusion-curriculum-dpo`, `diffusion-spo-training`, `flux-dpo-training`, `diffusion-dora-training`, `diffusion-lokr-training`, `diffusion-oft-training`, `diffusion-boft-training`, `lycoris-framework-integration`, `consistency-model-training`, `lcm-lora-training`, `scot-training`, `deepcache-training-free-acceleration`, `streamdiffusion-realtime-generation-pipeline`, `hidiffusion-training-free-high-resolution-generation`, `senseflow-flow-matching-distillation`, and related modules. - **Data pipeline**: `data-loading-system`, `data-sharding`, `streaming-datasets`, `data-preprocessing`, `tokenization`, `packing-for-efficiency`, `sequence-padding`, `dynamic-batching`, `data-augmentation`, `text-augmentation`, `back-translation`, `paraphrasing`, `data-filtering`, `quality-scoring`, `deduplication`, `data-mixing`, `curriculum-design`, `data-selection`. ### V2 Anti-Cheat Classifier Training Contract This is a concrete, domain-specific training contract for the V2 fighting-game project. It trains four classifiers (`smurf`, `win-trading`, `coordinated-throw`, `geographic-anomaly`) from event-bus labeled data. The contract enforces structural completeness before any training begins — a missing classifier, dataset, or label class causes `buildNousV2AntiCheatTrainingPlan` to throw `NousV2AntiCheatTrainingPlanError`. The `mayInfluenceRollback: false` field is fixed and non-overridable, ensuring trained classifiers can never affect the deterministic gameplay simulation path. Source: `libs/nous/training/src/v2-anti-cheat-classifier-training.ts`. ```typescript const NOUS_TRAINING_PACKAGE_NAME = '@nous/training'; const NOUS_V2_ANTI_CHEAT_TRAINING_SOURCE_OF_TRUTH_ID = 'nous.training.v2.anti-cheat-classifier-suite'; const NOUS_V2_ANTI_CHEAT_CLASSIFIERS = [ 'smurf', 'win-trading', 'coordinated-throw', 'geographic-anomaly', ] as const; type NousV2AntiCheatClassifierId = (typeof NOUS_V2_ANTI_CHEAT_CLASSIFIERS)[number]; type NousV2AntiCheatLabel = 'clean' | 'suspected' | 'confirmed'; type NousV2AntiCheatTrainingStage = | 'event-bus-ingest' | 'feature-extraction' | 'offline-training' | 'threshold-calibration' | 'human-review-shadow'; interface NousV2AntiCheatTrainingPlan { readonly schemaVersion: 1; readonly sourcePackageName: '@nous/training'; readonly sourceOfTruthId: typeof NOUS_V2_ANTI_CHEAT_TRAINING_SOURCE_OF_TRUTH_ID; readonly suiteId: string; readonly generatedAt: string; readonly classifierIds: readonly NousV2AntiCheatClassifierId[]; readonly datasets: readonly NousV2AntiCheatClassifierDataset[]; readonly labeledEvents: readonly NousV2AntiCheatLabeledEvent[]; readonly eventBusTopics: readonly string[]; readonly trainingLoopOwnerPackageName: '@nous/training'; readonly usesEventBusLabeledData: true; readonly modelCardRequired: true; readonly modelCardOwnerPackageName: '@nous/safety'; readonly mayInfluenceRollback: false; readonly minimumPrecision: number; // default 0.99 readonly minimumRecall: number; // default 0.75 readonly stages: readonly NousV2AntiCheatTrainingStage[]; readonly diagnostics: NousV2AntiCheatTrainingDiagnostics; } function buildNousV2AntiCheatTrainingPlan( input: NousV2AntiCheatTrainingPlanInput ): NousV2AntiCheatTrainingPlan; function validateNousV2AntiCheatTrainingPlanInput( input: NousV2AntiCheatTrainingPlanInput ): string[]; class NousV2AntiCheatTrainingPlanError extends Error { readonly validationErrors: readonly string[]; } ``` `NOUS_V2_ANTI_CHEAT_CLASSIFIER_EVENT_TOPICS` maps each classifier to required `v2.*` event-bus topics, e.g. `smurf` → `['v2.match.completed', 'v2.player.skill-profile.updated', 'v2.account.linked']`. `buildNousV2AntiCheatTrainingPlan` throws `NousV2AntiCheatTrainingPlanError` when any classifier, dataset, event topic, or label class (`clean` plus at least one non-`clean`) is missing. The plan's five stages run `event-bus-ingest → feature-extraction → offline-training → threshold-calibration → human-review-shadow`; `mayInfluenceRollback` is fixed `false`. --- ## 17. Safety and Governance (`@nous/safety`) `@nous/safety` hosts safety metadata and governance records for deployed AI systems. It is the package that makes a Nous-served model _accountable_ — every model deployed in the V2 project has a Model Card hosted here that documents its risk class, evaluation results, disclosure obligations, and fallback behaviour. The package also ships a broad set of classifiers and enforcement modules that consuming domains (Kuanyin, Themis) use to build safety policies around. The boundary is explicit: `@nous/safety` does _not_ own player settings, product rollback logic, or enforcement decisions. It produces records and classifier outputs; downstream domains decide what to do with them. Source: `libs/nous/safety/src/`, 64 modules. ### Module Groups The 64 modules span six safety and governance areas: - **Content classification / moderation**: `content-classifier`, `toxicity-detection`, `hate-speech-detection`, `harassment-detection`, `violence-detection`, `sexual-content-detection`, `self-harm-detection`, `spam-detection`, `abuse-detection`, `bot-detection`, `coordinated-behavior-detection`, `influence-operation-detection`, `misinformation-detection`, `manipulation-detection`, `synthetic-content-detection`, `deepfake-detection`, `watermark-detection`, `content-fingerprinting`. - **Injection / input safety**: `adversarial-input-detection`, `prompt-injection-detection`, `jailbreak-detection`, `code-injection-detection`, `sql-injection-detection`, `xss-detection`, `secrets-detection`, `input-validation`, `output-validation`. - **PII**: `pii-detection`, `pii-redaction`. - **Output enforcement**: `format-enforcement`, `length-enforcement`, `style-enforcement`, `topic-enforcement`, `safety-filtering`, `relevance-filtering`. - **Uncertainty / verification**: `abstention`, `confidence-estimation`, `calibration-analysis`, `uncertainty-quantification`, `selective-prediction`, `consistency-checking`, `factuality-checking`, `hallucination-detection`, `error-detection`, `external-verification`, `self-correction`, `explanation-generation`. - **Interpretability**: `interpretability-tools`, `attribution-analysis`, `attribution-extraction`, `goal-inference`, `intent-detection`, `preference-modeling`, `value-alignment-system`. - **Oversight / process**: `human-in-the-loop`, `oversight-mechanisms`, `approval-workflows`, `escalation-system`, `audit-trails`, `provenance-tracking`. - **Formal verification**: `kalika-formal-verification` — imports `@kalika/core` (`DerivationStep`, `Expr`, `ProvenResult`) and `@kalika/formal-verification` (`verifyKalikaComputationWithLean`, `LeanReplSession`, …) and wraps `ExternalVerifier` to produce typed safety invariant results (`KalikaFormalSafetyInvariantKind` ∈ `safety-constraint | invariant-preservation | policy-compliance | output-contract`). ### V2 Model Card Hosting The V2 Model Card system records governance metadata for each AI system deployed in the V2 project. `buildV2AIModelCard` is the generic builder; the three pre-built wrappers (`buildV2AdaptiveAIModelCard`, `buildV2AICommentaryModelCard`, `buildV2GenerationPipelinesModelCard`) produce cards for specific V2 AI systems. All three ship with `riskClass: 'limited'`, concrete `sourcePackages`, EU Article 50 disclosure language in `transparencyDisclosure`, and `evaluationResults` that reference the benchmarks backing the system. `buildV2AIModelCard` throws `V2ModelCardHostingError` on any missing required field, invalid risk class, or empty source/evaluation lists — incomplete Model Cards are never created silently. Source: `libs/nous/safety/src/v2-model-card-hosting.ts`. ```typescript const NOUS_SAFETY_PACKAGE_NAME = '@nous/safety'; const V2_ADAPTIVE_AI_MODEL_CARD_ID = 'v2.adaptive-ai-director.model-card'; const V2_ADAPTIVE_AI_MODEL_CARD_PATH = 'V2/docs/ai/model-cards/adaptive-ai-director.md'; const V2_AI_COMMENTARY_MODEL_CARD_ID = 'v2.ai-commentary.model-card'; const V2_AI_COMMENTARY_MODEL_CARD_PATH = 'V2/docs/ai/model-cards/ai-commentary.md'; const V2_GENERATION_PIPELINES_MODEL_CARD_ID = 'v2.generation-pipelines.model-card'; const V2_GENERATION_PIPELINES_MODEL_CARD_PATH = 'V2/docs/ai/model-cards/generation-pipelines.md'; type V2AIActRiskClass = 'minimal' | 'limited' | 'high' | 'prohibited'; interface V2AIModelCardEvaluationResult { readonly metric: string; readonly value: string; readonly threshold: string; readonly status: 'pass' | 'watch' | 'fail'; } interface V2AIModelCard { readonly schemaVersion: 1; readonly sourcePackageName: '@nous/safety'; readonly modelCardId: string; readonly serviceName: string; readonly owner: string; readonly riskClass: V2AIActRiskClass; readonly purpose: string; readonly sourcePackages: readonly string[]; readonly transparencyDisclosure: string; readonly optOutMechanism: string; readonly fallbackBehavior: string; readonly regionRestrictions: readonly string[]; readonly evaluationResults: readonly V2AIModelCardEvaluationResult[]; readonly hostedPath: string; readonly hostedBy: '@nous/safety'; readonly generatedAtUnixMs: number; readonly publicationStatus: 'hosted'; } function buildV2AIModelCard(input: V2AIModelCardInput): V2AIModelCard; function buildV2AdaptiveAIModelCard( overrides?: Partial ): V2AIModelCard; function buildV2AICommentaryModelCard( overrides?: Partial ): V2AIModelCard; function buildV2GenerationPipelinesModelCard( overrides?: Partial ): V2AIModelCard; class V2ModelCardHostingError extends Error { readonly validationErrors: readonly string[]; } ``` The three pre-built cards (Adaptive AI Director, AI Commentary, Generation Pipelines) are all `riskClass: 'limited'`, carry concrete `sourcePackages`, `transparencyDisclosure`, `optOutMechanism`, `fallbackBehavior`, `regionRestrictions` (EU Article 50 disclosure), and `evaluationResults`. `buildV2AIModelCard` throws `V2ModelCardHostingError` on missing required fields, an invalid `riskClass`, or empty `sourcePackages`/`evaluationResults`. ### V2 Anti-Cheat Classifier Model Card Source: `libs/nous/safety/src/v2-anti-cheat-classifier-model-card.ts`. ```typescript const V2_ANTI_CHEAT_CLASSIFIER_MODEL_CARD_ID = 'v2.anti-cheat-classifier-suite.model-card'; const V2_ANTI_CHEAT_CLASSIFIER_MODEL_CARD_PATH = /* V2/docs/ai/model-cards/... */; const V2_ANTI_CHEAT_MODEL_CARD_CLASSIFIERS = /* classifier id list */; type V2AntiCheatModelCardClassifierId = …; interface V2AntiCheatClassifierModelCard extends V2AIModelCard { … } function buildV2AntiCheatClassifierModelCard(...): V2AntiCheatClassifierModelCard; class V2AntiCheatClassifierModelCardError extends Error { … } ``` This pairs the `@nous/training` anti-cheat classifier suite with a hosted, limited-risk Model Card whose evaluation surface enforces review-only, appealable, off-rollback operation. --- ## 18. Generative Media Packages Eight specialised packages extend the generative-media surface beyond what `@nous/vision` provides. Each is an implemented TypeScript package with per-module `*.spec.ts` tests and a typed-class API. The packages are independent of each other — a consuming domain can import only the ones it needs. Module names below are taken directly from each `src/index.ts` barrel file. ### `@nous/image-control` (50 modules) Conditioned image-generation control: `controlnet-plus-plus-cycle-consistency`, `anydoor-reference-object-insertion`, `attend-and-excite-semantic-binding`, `boxdiff-layout-control`, `concept-sliders-lora-directions`, `attention-based-region-isolation`, `attention-heat-map-visualization`, `batch-instruction-editing-pipeline`, `combined-editing-pipeline`, `control-backend-comparison-harness`, `control-condition-preprocessor-registry`, `control-condition-quality-scorer`, and related modules. ### `@nous/video-control` (43 modules) Camera/subject control for video diffusion: `anyv2v-video-editing-pipeline`, `camera-trajectory-preset-library`, `camera-trajectory-visualization-workbench`, `depthdirector-depth-conditioned-camera-control`, `hunyuancustom-*` (dual-stream processing, multi-subject/multimodal generation, subject-aware temporal coherence, subject-identity preservation), `keyframe-based-video-editing-workflow`, `motion-prompting-camera-control-pipeline`, and related modules. ### `@nous/video-removal` (48 modules) Object removal / video inpainting (`package.json` depends on `bullmq` for job queuing): `cogvideox-fun-base-model`, `diffueraser-integration`, `avid-integration`, `grey-mask-generation`, `edge-artifact-detector`, `background-reconstruction-quality`, `automated-quality-gate`, `human-evaluation-pipeline`, `gemini-physics-consequence-analyzer`, `humoto-synthetic-data-pipeline`, `fine-tuning-quality-validation`, and related modules. ### `@nous/video-to-audio` (21 modules) Video/image-conditioned audio + Foley: `any2audio-*` (audio-editing, image-to-audio, text-to-audio, multimodal-conditioning, model-loader, quality-metrics pipelines), `av-link-*` (inference pipeline, model loader), `foley-*` (intensity control, library augmentation, A/B comparison), `audio-video-sync-quality-validation-pipeline`, and related modules. ### `@nous/advanced-speech` (31 modules) Advanced TTS / voice cloning / music: `f5-tts-model-loader`, `cosyvoice-integration`, `fish-audio-s2-pro-integration`, `cross-tts-voice-cloning`, `accent-conversion`, `emotion-prosody-control-api`, `multi-language-tts-routing`, `music-generation-evaluation-suite`, `music-generation-midi-control`, `music-to-music-style-transfer`, `music-continuation-style-transition`, and related modules. ### `@nous/generative-relighting` (23 modules) Image/video/3DGS relighting: `ic-light-*` (foreground relighting, background generation, model loader), `genlit-video-relighting-pipeline`, `gaussctrl-3dgs-editing-pipeline`, `ctrl-d-dynamic-3dgs-editing-pipeline`, `intergsedit-interactive-3dgs-editing-api`, `intrinsic-edit-pipeline`, `environment-map-conditioning`, `lightlab-light-source-control-pipeline`, `lumigauss-material-decomposition-pipeline`, and related modules. ### `@nous/portrait-animation` (23 modules) Audio/video-driven portrait animation and dubbing: `liveportrait-*` (expression-transfer, stitching-animation), `hallo3-*` (portrait-animation inference, video-DiT model loader), `echomimic-audio-to-facial-motion-pipeline`, `chatanyone-realtime-portrait-pipeline`, `im-portrait-3d-aware-video-diffusion`, `multi-person-portrait-animation-pipeline`, `cross-language-dubbing-pipeline`, `dubbing-batch-processor`, `dubbing-quality-validation-pipeline`, and related modules. ### `@nous/joint-av-generation` (34 modules) Joint audio-video generation with physics awareness: `joint-av-generation-library`, `joint-av-generation-manifest`, `joint-denoising-pipeline`, `dual-branch-mmdit-architecture`, `denoising-temporal-alignment-validator`, `camera-language-planner`, and physics-awareness modules (`collision-contact-physics`, `fabric-material-dynamics`, `fluid-dynamics-awareness`, `gravity-simulation-awareness`, `inertia-momentum-preservation`). --- ## 19. Concordia Cooperative-Intelligence Packages Four packages provide the computational layer for multi-party bargaining and dispute resolution used by the Concordia domain. The boundary is important: these packages own _algorithms_ (preference model fitting, multi-objective search, privacy primitives) but not _policy_ (legal boundaries, settlement authority, product UX). Concordia provides the contracts (`@concordia/contracts`) that define parties, cases, and outcomes; Nous provides the math and privacy machinery that operates on those contracts. All four packages have implemented source but no `*.spec.ts` files yet. All three bargaining packages depend on `@concordia/contracts` (except `@nous/concordia-sealed-memory`, which depends only on `zod`). All use `zod` for schema validation. Their `src/index.ts` doc comments reference phase numbers §179.x. ### `@nous/preference-inference` (24 modules) Source: `libs/nous/preference-inference/src/`. Pairwise preference elicitation and five calibrated utility-model estimators (per `index.ts` doc comment): Bradley-Terry, Thurstone-Mosteller, Plackett-Luce, Gaussian-process preference (Chu-Ghahramani 2005), and a neural utility ranker. Exported surfaces: - **Elicitation**: `buildElicitationPlan`, `computeTargetPairCount`, `meetsStabilityThresholds`, `PairwiseElicitationPlanSchema`, `StabilityThresholdsSchema`, `InferenceModelFamilySchema`. - **Comparison prompts**: `buildPairwiseComparisonPrompt`, `buildPairwisePromptFromStatements`, `parsePairwiseComparisonResponse`, `isDecisive`, with `ComparisonChoiceSchema`, `AbstentionReasonSchema`, `CitationSchema`, `PairwiseComparisonResponseSchema`. - **Estimators**: `fitBradleyTerry`, `fitThurstoneMosteller`, `fitPlackettLuce`, `fitGaussianProcessPreference`, `fitNeuralRanker`, `fitUtilityModel`, plus `toContractUtilityModel`, `buildCredibleIntervals`. - **Calibration / stability**: `calibrationMetrics`, `predictiveCalibration`, `aggregateStability`, `runStabilityProbes`, prompt-perturbation builders (`buildParaphrasePerturbation`, `buildOrderSwapPerturbation`, `buildAdversarialFramingPerturbation`, …). - **Active learning**: `computeParetoFrontier`, `selectActiveLearningPairs`, `selectStabilityProbes`. - **Utility scoring / gating**: `buildUtilityScores`, `gatePairwiseComparison`, `observationsForOptimizer`, `assertOptimizerSafe`, `summarizeGatedComparisons`. - **Multi-attribute utility**: `computeMultiAttributeUtility`, `scoreCandidates`, `inferIssueWeights`. - **Hard constraints**: `checkFeasibility`, `filterFeasibleCandidates`, `guardedUtility`, `reservationPointConstraint`, `authorityCeilingConstraint`, `redlineConstraint`. - **BATNA**: `analyzeBATNA`, `evaluateAcceptance`, `computeBatnaPlausibility`. - **Fairness**: `nashProduct`, `utilitarianSum`, `maxMinUtility`, `egalitarianWelfare`, `kalaiSmorodinskyDistance`, `envy`, `regret`, `inequality`, `burdenSymmetry`, `proceduralDignity`, `computeAllFairnessMetrics`, plus `BUILTIN_PROFILES`, `selectFairnessProfile`, `applyFairnessProfile`, `detectProtectedClassViolations`. ### `@nous/agreement-search` (22 modules) Source: `libs/nous/agreement-search/src/`. Search-kernel registry plus seven implemented search kernels over a candidate-agreement space: - **Registry**: `SEARCH_KERNEL_REGISTRY`, `getKernelDescriptor`, `rankKernelsForCaseProfile`, with `SearchKernelDescriptorSchema`, `SearchKernelCapabilitiesSchema`, `SearchKernelCostProfileSchema`. - **Search kernels**: `runNashGeneticSearch` (Nash-product genetic search), `runNsgaIISearch` (NSGA-II, with `fastNonDominatedSort`, `crowdingDistance`), `runMapElitesSearch`, `runMctsSearch` (MCTS/LATS), `runCpSatSearch` (CP-SAT), `runBayesianOptimization` (with `expectedImprovement`, `upperConfidenceBound`, `probabilityOfImprovement`), `runPsroSearch` (PSRO opponent modeling), and `runCoalitionStabilitySearch` (Shapley values exact and Monte-Carlo, least-core allocation, bargaining set). - **Candidate generation / mutation**: `generateSeedCandidates`, ten domain playbooks (`procurementPlaybook`, `creativeRoyaltyPlaybook`, `daoGovernancePlaybook`, `platformModerationAppealPlaybook`, `marketplaceDisputePlaybook`, `cofounderEquityPlaybook`, `supplierServiceLevelPlaybook`, `collaborativeProductionPlaybook`, `multiplayerGuildGovernancePlaybook`, `agentToAgentContractPlaybook`), ten clause mutators (`numericPerturbation`, `deadlineShift`, `proportionalSplit`, `installmentSchedule`, `equityVesting`, `royaltyWaterfall`, `scopeNarrowing`, `escalationLadder`, `auditRight`, `reversibleTrialPeriod`), and proposal decomposition/recombination (`decomposeProposal`, `recombineProposals`, `crossoverPlan`). - **GA operators**: seeded RNG (`mulberry32`), `crossoverCandidates`, `applyRandomMutations`, `produceChild`. - **Audits / explanations**: `auditSpecGaming`, `auditCoerciveChoice`, `filterCandidates`, `renderSharedNeutralExplanation`, `renderPartyPrivateBriefing`, `renderParetoFrontierExplanation`, `computeSteeringBias` / `createControlRegistry` (workbench controls). ### `@nous/cooperative-bargaining` (2 modules) Source: `libs/nous/cooperative-bargaining/src/`. A session-scoped bargaining substrate bridging `@concordia/contracts` and the agreement-search kernels. ```typescript const BargainingPhaseSchema = z.enum([ 'intake', 'preference_inference', 'generation', 'scoring', 'frontier_stable', 'review_pending', 'awaiting_acceptance', 'closed', ]); const BargainingClosureReasonSchema = z.enum([ 'accepted', 'withdrawn_by_party', 'escalated_to_reviewer', 'boundary_triggered', 'budget_exhausted', 'cancelled_by_mediator', ]); const BargainingBudgetSchema = z.object({ maxIterations: z.number().int().positive(), maxWallClockSeconds: z.number().int().positive(), maxModelSpendMicros: z.string().regex(/^\d+$/), // USD micros, string-encoded }); ``` `StakeholderSchema` projects `@concordia/contracts` `Party` to the hot-path fields kernels need (`partyId`, `displayName`, `role`, `authorityVerificationState` ∈ `unverified | self_attested | reviewer_verified | documentary_verified | revoked | expired`, `canAccept`, `canBindOrganization`). Exported functions: `buildBargainingSession`, `advancePhase`, `closeSession`, `recordIteration`, `allStakeholdersCanAccept`, `stakeholdersBlockingAcceptance`, `toStakeholder`. ### `@nous/concordia-sealed-memory` (5 modules) Source: `libs/nous/concordia-sealed-memory/src/`. Concordia privacy primitives: - **Sealed-memory store** (`sealed-memory-store.ts`): envelope-encrypted store addressed by reference, with a `KmsProvider` abstraction (`createInMemoryKmsProvider`), per-operation authorization (`AuthorizeFn`), and an audit sink. `SealedMemoryOperation`, `SealedMemoryRole`, `AccessActor`, `AuditRecord`, `AuditOutcome` are exported types. - **Zero-retention routing** (`zero-retention-mode.ts`): `chooseEndpoint`, `requiredRetentionMode`, `useCaseRequiresZeroRetention`, `generateRetentionAttestation`, `verifyAttestation`, `createHmacAttestationSigner`, with `RetentionMode`, `ZeroRetentionRoutingDecision`, `LocalModelEndpoint` types. - **Confidential compute** (`confidential-compute.ts`): `createConfidentialScoringSession`, `runConfidentialScoring`, `signTeeAttestation`, `verifyTeeAttestation`, with `IsolationTier`, `TeeAttestation`, `TeeAttestationPolicy`, `ConfidentialScoringDecision` types. --- ## 20. Configuration and Environment ### Configuration Objects Nous has **no single domain-wide `NousConfig` object**. This is a deliberate consequence of the composable-primitive design principle: a consumer that only needs embedding generation should not pay for configuration schema validation across inference, batching, agent, and safety modules it does not use. Each module is configured through its own `*Options`/`*Config` argument, documented in the sections above (e.g. `OnnxInferenceEngineOptions`, `OnnxSessionManagerOptions`, `DynamicBatchingOptions`, `KvCacheManagerOptions`, `CpuExecutionProviderConfig`, `QuantizationConfig`, `EmbeddingGeneratorOptions`, `CompletionAPIOptions`, `AgentArchitectureOptions`). The consumer constructs and wires only the objects it needs. ### Environment Variables Nous reads environment variables only for **hardware-capability detection** in the execution-provider modules — there are no `NOUS_DEFAULT_MODEL`, `NOUS_KV_CACHE_MAX_TOKENS`, or similar runtime-behaviour overrides. The `NOUS_SAFETY_PACKAGE_NAME` and `NOUS_TRAINING_PACKAGE_NAME` identifiers are exported TypeScript _constants_, not environment variables. The complete list of `NOUS_*` variables referenced in source is: | Variable | Provider module | Purpose | | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | -------------------------------- | | `NOUS_CUDA_DEVICE_COUNT` | cuda | CUDA device-count override | | `NOUS_CUDA_COMPUTE_CAPABILITY` | cuda | CUDA compute-capability override | | `NOUS_ROCM_DEVICE_COUNT` | rocm | ROCm device-count override | | `NOUS_DML_AVAILABLE` | dml | Force DirectML availability | | `NOUS_DML_ADAPTER_COUNT` | dml | DirectML adapter-count override | | `NOUS_DML_FP*`, `NOUS_DML_METACOMMANDS` | dml | DirectML feature overrides | | `NOUS_METAL_AVAILABLE` | metal | Force Metal availability | | `NOUS_METAL_WORKING_SET_BYTES` | metal | Metal working-set budget | | `NOUS_COREML_AVAILABLE` | coreml/npu | Force CoreML availability | | `NOUS_APPLE_NEURAL_ENGINE` | npu | Apple Neural Engine override | | `NOUS_WEBGPU`, `NOUS_WEBGPU_ADAPTER_TYPE`, `NOUS_WEBGPU_FP*`, `NOUS_WEBGPU_TIMESTAMP_QUERY` | webgpu | WebGPU capability overrides | | `NOUS_VULKAN_AVAILABLE`, `NOUS_VULKAN_FP*`, `NOUS_VULKAN_INT*`, `NOUS_VULKAN_GPU_CLASS`, `NOUS_VULKAN_QUEUE_FAMILIES`, `NOUS_VULKAN_TIMELINE_SEMAPHORE` | vulkan | Vulkan capability overrides | | `NOUS_NNAPI_AVAILABLE` | nnapi | Force NNAPI availability | | `NOUS_QNN_AVAILABLE`, `NOUS_QNN_HTP` | qnn | QNN capability overrides | | `NOUS_NPU_AVAILABLE` | npu | Force NPU availability | `detectExecutionProviderEnvironment` (Section 4) additionally reads standard toolchain variables — `CUDA_PATH`, `CUDA_HOME`, `CUDA_VISIBLE_DEVICES`, `NVIDIA_VISIBLE_DEVICES`, `ROCM_HOME`, `HIP_PATH`, `VULKAN_SDK`, `ANDROID_ROOT`, `QNN_SDK_ROOT`, and `LD_LIBRARY_PATH`. `NOUS_SAFETY_PACKAGE_NAME` and `NOUS_TRAINING_PACKAGE_NAME` are exported **constants** (the package name strings), not environment variables. There are no `NOUS_DEFAULT_MODEL`, `NOUS_PREFERRED_PROVIDERS`, `NOUS_KV_CACHE_MAX_TOKENS`, `NOUS_MODEL_REGISTRY_PATH`, or `NOUS_LOG_LEVEL` variables in the codebase. --- ## 21. Integration Points This section summarises the declared integration boundaries: what Nous depends on, and how consuming domains wire into it. Engineers adding a new cross-domain dependency should update the table in §21.2 and justify the dependency boundary. ### Optional Runtime Dependency `onnxruntime-node` (`^1.20.0`) is declared as an `optionalDependency` of `@nous/core` only. `loadOnnxRuntime` (Section 3) loads it lazily and throws `OnnxRuntimeNotAvailableError` when it is absent, so any consumer can compile `@nous/core` without the native runtime installed. A custom runtime module can be supplied through `OnnxRuntimeLoaderOptions.load` or `OnnxInferenceEngineOptions.runtime`. ### Cross-Domain Dependencies The following workspace dependencies are the only cross-domain `package.json` entries across all nineteen implemented packages. Any other external symbols visible in a `@nous/*` package are brought in through `devDependencies` (e.g. `@types/node`) and do not create a runtime dependency. The reason for each boundary is explained in `architecture.md` §12. | `@nous` package | Depends on (workspace) | | ------------------------------- | --------------------------------------------- | | `@nous/safety` | `@kalika/core`, `@kalika/formal-verification` | | `@nous/preference-inference` | `@concordia/contracts` | | `@nous/agreement-search` | `@concordia/contracts` | | `@nous/cooperative-bargaining` | `@concordia/contracts` | | `@nous/concordia-sealed-memory` | (none beyond `zod`) | These are the only declared cross-domain library dependencies in any `libs/nous/*/package.json`. `@concordia/contracts` and `@kalika/*` belong to the Concordia and Kalika domains respectively. ### V2 Composition Boundary The V2 contracts in `@nous/training` and `@nous/safety` are designed for composition by V2-layer packages. The training plan fixes `modelCardOwnerPackageName: '@nous/safety'` and `mayInfluenceRollback: false`; the model cards fix `hostedBy: '@nous/safety'`. Nous owns the training loop and the Model Card record; product rollback/enforcement behaviour is owned outside Nous. --- ## 21a. V2 Cross-Domain Service Contracts The V2 fighting-game project composes the Nous libraries described above into shipping services under `V2/services/`. Each service is a thin, deterministic composition layer: it imports the Nous primitives, fixes the cross-domain ownership fields, and exposes a single validated surface object. None of these services may touch the rollback netcode path — every surface fixes `mayInfluenceRollback: false`, and the rollback simulation never opens an RPC to a classifier, a Model Card host, or a dispute substrate. This subsection documents the four cross-domain boundaries that consume Nous from V2. ### V2 Anti-Cheat Classifier Contract (`@v2/nous-anti-cheat-classifiers`) The **V2 Anti-Cheat Classifier Contract** is the composition surface that pairs the `@nous/training` classifier suite (§16) with its hosted `@nous/safety` Model Card (§17). The service package `@v2/nous-anti-cheat-classifiers` depends on both `@nous/training` and `@nous/safety` at `workspace:*` and exposes `buildV2NousAntiCheatClassifierSuite`, which builds the training plan and the Model Card together, then cross-validates them before returning a single `V2NousAntiCheatClassifierSuite`. The four classifiers are fixed and domain-specific: `smurf`, `win-trading`, `coordinated-throw`, and `geographic-anomaly`. The suite trains them off-rollback from event-bus labeled data (`usesEventBusLabeledData: true`, `v2PublishesLabeledTrainingData: true`) and surfaces inference only as a review aid — never as automatic enforcement. The contract pins `humanReviewRequired: true`, `appealsRequired: true`, `automatedDisciplineAllowed: false`, and `deprecatedMlPipelineAbsent: true`, so the deprecated `@nous/ml-pipeline` path can never be reintroduced. Inference output feeds first-line moderation as evidence; a human reviewer makes every disciplinary decision, and every decision is appealable. `buildV2NousAntiCheatClassifierSuite` throws `V2NousAntiCheatClassifierSuiteError` (with a populated `validationErrors` array) when a classifier is missing from either the training plan or the Model Card, when the training loop is not owned by `@nous/training`, or when the Model Card is not hosted by `@nous/safety`. The suite also emits an `anti_cheat` audit-publication request (see "Audit Publication" below) so that every review event lands in the retained Oshun audit ledger. Source: `apps/v2/nous-anti-cheat-classifiers/src/nous-anti-cheat-classifiers.ts`. ### V2 Adaptive AI Director and EU AI Act Surface Nous is one of three domains behind the **V2 Adaptive AI Director** EU AI Act conformity surface (`@v2/eu-ai-act-surface`). Nous owns the _Model Card_ leg: `@nous/safety` hosts the limited-risk cards for every shipping V2 AI system — the Adaptive AI Director (`adaptive-ai-director.md`), AI commentary (`ai-commentary.md`), the generation pipelines (`generation-pipelines.md`), and the anti-cheat classifier suite. `buildV2AdaptiveAIModelCard`, `buildV2AICommentaryModelCard`, `buildV2GenerationPipelinesModelCard`, and `buildV2AntiCheatClassifierModelCard` (§17) all set `hostedBy: '@nous/safety'` and `publicationStatus: 'hosted'`. The other two legs sit outside Nous: `@psyche/action-safety` produces the player-facing classifier transparency notice and opt-out, and `@themis/accountability` registers each Model Card as an AI-system-of-record and owns the regulator-facing audit export. The composition service binds the three together (`modelCardHostedBy: '@nous/safety'`, `auditExportOwnedBy: @themis/accountability`) while keeping the whole surface off the deterministic rollback path. The anti-cheat Model Card additionally links `@themis/accountability` and `@themis/dispute-resolution` so that appeals are routed to a real dispute workflow rather than handled implicitly. ### V2 Concordia Cooperative-Intelligence Substrate (`@v2/concordia-substrate`) The Concordia cooperative-intelligence packages (§19) are composed by `@v2/concordia-substrate` into the dispute-resolution substrate that powers anti-cheat appeals, tournament-result disputes, and crew conflicts. The service wires all four Nous primitives — `@nous/cooperative-bargaining` (the bargaining session and acceptance ledger), `@nous/preference-inference` (Bradley–Terry elicitation over disputed terms), `@nous/agreement-search` (Nash genetic search over the candidate-agreement space), and `@nous/concordia-sealed-memory` (the party-isolated, KMS-sealed intake store) — alongside `@iris/concordia-assistant` for appellant/arbiter dialogue, `@oshun/concordia-integration` for event routing, and `@kuanyin/concordia-restorative` for the safety-gated restorative branch. The whole substrate is flag-gated behind `ENABLE_V2_CONCORDIA_SUBSTRATE` and fixes `offRollback: true` / `mayInfluenceRollback: false`. Sealed-memory keeps each party's intake private and exposes only sealed references, so an arbiter never sees the opposing party's context. The Nous side of this boundary is unchanged from §19: Nous ships reusable preference/search/bargaining/privacy primitives, and Concordia (through the V2 substrate) owns the dispute policy and settlement authority. Source: `apps/v2/concordia-substrate/src/concordia-substrate.ts`. ### Audit Publication through `@oshun/audit-platform` The anti-cheat service does not retain its own audit log. Instead, **V2 publishes** canonical audit events through `@oshun/audit-platform`, and **Oshun retains and exports** them. The shared `@oshun/audit-platform` publisher accepts three V2 publication kinds — `moderation`, `anti-cheat`, and `DSR` (data-subject request) — and `@v2/nous-anti-cheat-classifiers` emits the `anti_cheat` kind via `buildV2AuditPublicationRequest({ kind: 'anti_cheat', … })` on the canonical `v2.anti_cheat.review_published` action. The publication is tagged with the `v2-anti-cheat-audit-retention` policy and carries `v2Publishes: true` / `oshunRetainsAndExports: true`, so a single Oshun-owned ledger holds the retained record and serves the regulator/investigation export. This is the same routing used by Kuanyin first-line moderation and the Themis privacy DSR service; the cross-domain contract is documented in `V2/docs/integration/v2-audit-platform-routing.md`. --- ## 22. Acceptance Criteria These criteria apply to any pull request that touches `libs/nous/`. They encode the architectural invariants described in this document and in `architecture.md`. A reviewer should check each criterion independently — passing the test suite alone does not satisfy all of them. A change to the Nous domain is acceptable when: 1. **Build and typecheck pass.** `tsc --noEmit -p libs/nous//tsconfig.lib.json` succeeds for every touched package, and the `@nx/esbuild:esbuild` build target produces ESM output. 2. **Tests pass.** Every implemented module in `core`, `llm`, `vision`, `audio`, `training`, `safety`, and the eight generative-media packages has a sibling `*.spec.ts`; `vitest` must pass for touched packages. New modules in those packages must add a sibling spec. 3. **Types are exhaustive.** New domain objects use discriminated literal unions (not loose strings) for state/kind fields, mirroring `OnnxExecutionProviderName`, `AgentLifecycleState`, `AgentTaskState`, `BargainingPhaseSchema`, and `NousV2AntiCheatLabel`. 4. **Provider abstraction holds.** Inference code routes through `resolveExecutionProviders` / `OnnxSessionManager`; no module hard-codes a single backend or bypasses provider fallback. 5. **The runtime stays optional.** `@nous/core` continues to compile and import without `onnxruntime-node`; the runtime is reached only via `loadOnnxRuntime`. 6. **V2 contracts validate strictly.** `buildNousV2AntiCheatTrainingPlan` and `buildV2AIModelCard` (and the wrappers) reject incomplete input by throwing their typed error classes with a populated `validationErrors` array. 7. **Concordia boundary is preserved.** `@nous/preference-inference`, `@nous/agreement-search`, `@nous/cooperative-bargaining`, and `@nous/concordia-sealed-memory` expose only reusable preference/search/bargaining/privacy primitives; domain policy and settlement authority remain in the Concordia domain. 8. **Barrel exports stay complete.** New modules are re-exported from the owning package's `src/index.ts`. --- ## Document Provenance This specification was written against the source under `libs/nous/` at the state captured in the working tree. Type signatures, enum members, class methods, constants, and identifiers are quoted from: `core/src/types.ts`, `core/src/runtime-loader.ts`, `core/src/execution-providers.ts`, `core/src/cpu-execution-provider.ts`, `core/src/session-manager.ts`, `core/src/inference-engine.ts`, `core/src/model-loader.ts`, `core/src/model-registry.ts`, `core/src/dynamic-batching.ts`, `core/src/kv-cache.ts`, `core/src/quantization.ts`, `core/src/embedding-generation.ts`, `core/src/similarity-search.ts`, `core/src/{speculative-decoding,multi-lora-serving}.ts`, `llm/src/{completion-api,chat-completion-api,prompt-template-system,agent-architecture,document-ingestion}.ts`, `vision/src/{object-detection,stable-diffusion}.ts`, `training/src/v2-anti-cheat-classifier-training.ts`, `safety/src/{v2-model-card-hosting,v2-anti-cheat-classifier-model-card,kalika-formal-verification}.ts`, `cooperative-bargaining/src/bargaining-session.ts`, and the `src/index.ts` barrels of all nineteen implemented packages, plus every `package.json` and `project.json` under `libs/nous/`. Module-name lists are taken directly from the corresponding `src/index.ts` barrel files.