Eve library

phase-88.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 88: ML Sovereignty — Production Model Serving (SOTA April 2026)#

Integrate industry-leading inference engines and optimization techniques for serving self-hosted models at production throughput and latency.


88.1 LLM Serving Engines#

88.1.1 Engine Integration#

  • 88.1.1.1 Integrate vLLM v0.7+ as primary LLM serving engine (PagedAttention, continuous batching, optimized for Blackwell B200 — fastest TTFT across concurrency levels)
  • 88.1.1.2 Build vLLM Docker image for RunPod deployment with multi-LoRA support (serve N fine-tuned adapters on single base model using S-LoRA/LoRAX pattern)
  • 88.1.1.3 Integrate SGLang v0.4+ for structured generation workloads (RadixAttention prefix caching, constrained JSON/code output, cache-aware load balancer)
  • 88.1.1.4 Integrate TensorRT-LLM v1.2+ for NVIDIA-optimized inference (speculative decoding up to 3.6x, 1000 tok/s per user on Blackwell)
  • 88.1.1.5 Integrate llama.cpp with latest GGUF advances (imatrix quantization, AWQ-to-GGUF pipeline, sharding, speculative decoding, multimodal via libmtmd)
  • 88.1.1.6 Build model serving router that selects optimal backend per request (vLLM for chat, SGLang for structured output, TensorRT-LLM for throughput, llama.cpp for edge)
  • 88.1.1.7 Implement Ollama-compatible API endpoint for developer experience (local development parity with production)
  • 88.1.1.8 Build multi-tenant model isolation (per-customer LoRA adapters via LoRAX pattern with adapter hot-loading and memory-efficient sharing)

88.2 Inference Optimization#

88.2.1 Throughput & Latency Optimization#

  • 88.2.1.1 Implement speculative decoding pipeline with EAGLE-3 (fuse hidden states from multiple layers → 2-6x speedup)
  • 88.2.1.2 Build KV cache compression using Expected Attention and ClusterAttn (predict future attention → compress KV cache → lower memory without quality loss)
  • 88.2.1.3 Implement prefix caching with LMCache integration (up to 15x throughput, 2x lower latency for shared-prefix workloads)
  • 88.2.1.4 Build EXL3 quantization pipeline (ExLlamaV3 trellis-based encoding: 140GB → less than 40GB VRAM, 120 tok/s on 24GB GPU)
  • 88.2.1.5 Implement quantization format conversion pipeline (HuggingFace → GGUF imatrix, AWQ, GPTQ, EXL3, MXFP4 — automated with quality validation)
  • 88.2.1.6 Build Flash Attention 3 integration (async + low-precision + paged KV + incoherent FP8 processing)
  • 88.2.1.7 Implement dynamic batching optimizer (group requests by sequence length, maximize GPU utilization per batch)
  • 88.2.1.8 Build model warmup and preloading system (pre-populate KV cache for system prompts, keep hot models in GPU memory)
  • 88.2.1.9 Implement graceful model swapping (zero-downtime: load new version in background → swap atomically → unload old)
  • 88.2.1.10 Build inference cost calculator per endpoint (real-time dollars per token by model, backend, GPU, quantization)
  • 88.2.1.11 Implement request priority queuing (interactive < 500ms target, batch < 5s target, background — with SLA monitoring)
  • 88.2.1.12 Build Kubernetes-native KV cache-aware request routing (llm-d pattern: 87% cache hit rate, 88% faster TTFT for warm hits)
  • 88.2.1.13 Implement NVIDIA MIG partitioning for model serving (partition GPUs into isolated instances for concurrent model serving, auto-detect MIG-capable GPUs, configure partition profiles per model size, Kubernetes MIG-aware scheduling, up to 40% GPU cost reduction)
  • 88.2.1.14 Build optimized embedding serving pipeline (token-count-based batching for short queries, padding removal with variable-length concatenation, FP8 quantized embedding inference for 50%+ throughput, dedicated embedding endpoint, batch queue for bulk indexing)
  • 88.2.1.15 Build tiered model caching system (hot: GPU memory for high-traffic with <500ms TTFT, warm: SSD/NVMe for medium-traffic with <2s load, cold: S3/MinIO for archival — LRU eviction with SLA-aware pinning, predictive pre-loading)

88.2.2 Streaming, Function Calling & Structured Output#

  • 88.2.2.1 Build SSE streaming endpoint for all model serving backends (token-level streaming, data: message format per token, [DONE] completion event, auto-reconnect support, backpressure handling, streaming latency monitoring)
  • 88.2.2.2 Implement streaming with tool-use interruption (detect function call tokens mid-stream → pause generation → execute tool → inject result → resume, OpenAI-compatible function calling protocol)
  • 88.2.2.3 Build function calling / tool use serving layer (parse tool definitions, inject schemas into system prompts, detect/extract function call blocks, execute tools, inject results, resume — OpenAI-compatible and MCP-compatible protocols)
  • 88.2.2.4 Implement structured output serving with constrained decoding (JSON Schema enforcement via finite state machine token masking, grammar-based decoding for custom formats, 100% schema compliance in strict mode)

88.3 Model Format Management#

88.3.1 Conversion & Registry#

  • 88.3.1.1 Build automated model conversion pipeline (HuggingFace SafeTensors → GGUF + AWQ + GPTQ + EXL3 + ONNX + TensorRT → quality validation per format)
  • 88.3.1.2 Implement model format registry (track which formats exist for each model, quality metrics per format, storage locations)
  • 88.3.1.3 Build MXFP4 native quantization support (following GPT-OSS pattern for maximum efficiency)
  • 88.3.1.4 Implement automated imatrix calibration for GGUF quantization (importance-matrix generation from representative prompts)
  • 88.3.1.5 Build model sharding manager for large models (automatic split across multiple GPUs with optimal shard placement)

88.3.2 Unified Multi-Framework Serving#

  • 88.3.2.1 Integrate NVIDIA Dynamo-Triton (formerly Triton Inference Server) as unified multi-framework serving platform (ensemble model DAG pipelines, concurrent model execution, dynamic batching across TensorRT + PyTorch + ONNX + Python backends, HTTP/gRPC, model repository management)