Eve Small-Model Excellence (SMX) — Design (2026-08-16)

Companion execution ledger: EVE_SMALL_MODEL_EXCELLENCE_TODOS_2026-08-16.md (root). Status: DESIGN — nothing below is shipped unless a TODOS checkbox says so.

Thesis#

Run the assistant brain (member-facing Lilith + builder-facing Eve, one runner) on the smallest model that passes a regression-locked evaluation deck — today's champion is deepseek/deepseek-v4-flash-0731 at ~$0.10/$0.18 per 1M tokens, ~100x under frontier — and make that sustainable by moving intelligence out of the model and into the harness: a strong model (Claude, at build time, in session) does the capability analysis, writes the skills, designs the checkers, and distills the exemplars; the small model executes inside a harness that routes, scopes, validates, verifies, and escalates. The deck is the contract between the two: no skill line, prompt edit, checker, or model swap lands without a measured delta.

This is not a new philosophy for this repo — it is the Eve polish initiative's own discovery, systematized. The polish work already proved (with 20-run arms and Fisher exact tests) that a prompt sentence is a request and a server-side check is a mechanism (routes/assistant.ts:1143 block, EVE-VIS-128/277, the announce-before-act wire hold). SMX finishes the thought: compile-time intelligence, run-time economy.

Scope#

In scope: the assistant brain end to end — provider bindings (apps/oshun/bff/src/assistant/agent-provider-config.ts), the turn runner (agent-turn-runner.ts), prompt assembly and toolset assembly (routes/assistant.ts streaming turn route), all tool surfaces (member domain adapters, docs, tours, audit, admin, workbench read/write), the eval harness (evals/), turn metrics, and the OpenRouter routing configuration in @oshun/ai.

Out of scope: the codex coding-agent lane (tools/eve-codex-agent.mjs, subscription-billed, its own loop); the deterministic engine's message route (already model-free); voice STT/TTS vendors; member-web panel rendering (polish initiative owns it). The pi-ai subscription transport (CodexSubscriptionProvider) stays as an explicit-only provider and benefits from every harness change for free, since all of SMX lands above the LLMProviderInterface seam.

Adjacent, not forked: the repo has a second brain — @oshun/assistant, the Hermes-parity autonomous agent (channels, scheduling, its own Voyager-style skill store; docs/audits/HERMES_PARITY_TODOS.md). SMX builds for the BFF brain, but the deck harness pattern, skill disciplines, and checkers are designed so that stack can consume them too — two brains, one measurement doctrine.

Doctrine#

Eight principles. Each is anchored to evidence already in this repo; none is aspiration.

1. Mechanism over request. A rule the model must obey becomes a server-side check, not (only) a prompt sentence. Anchor: EVE-VIS-128 (audit-skip provenance enforced by the store), EVE-VIS-277 (mutating tools refuse without the confirm bridge), the announce-before-act stream hold, and the measured verdict "this sentence is NOT a fix" (20%→6%, p=0.34) in routes/assistant.ts:1143.
2. Evals decide, not vibes. Every prompt, skill, checker, and model change is judged by the deck, with pinned routing and stated statistics. Anchor: the repo's Fisher-exact discipline and the existing pure, unit-tested grader ("a vacuous grader cannot hide", evals/eval-harness.ts).
3. Compile-time intelligence, run-time economy. The strong model is a compiler: it analyzes failures, writes skills, authors exemplars, proposes checkers — at build time, in session, costing nothing at runtime. The small model is the runtime. The strong model NEVER sits in the serving path (CLAUDE.md cost rule).
4. Smallest model per LEG, not per product. Turn execution, routing, judging, and embedding are separate legs with separate bindings; each leg gets the cheapest model that passes its own gate. A product-wide "we use model X" is a category error.
5. Context is a budget, and small models pay it first. Long undifferentiated prompts are where flash-class models drown: today every tool-capable turn carries the full conduct block, the anchor registry, and the curated tour catalog regardless of task, and an admin turn can offer ~60 tools. Anchor: the measured tara_favorites skip (4/20) under the full surface.
6. Every failure becomes a case; every fix becomes a mechanism, a skill line, or an exemplar — traceable both ways. Anchor: the defect ledger (279 rows, every closed S1/S2 with a runnable lock, tools/eve-polish/ ledger-lock-audit.mjs). SMX extends the same discipline to model behavior.
7. Price the route, not the model. An OpenRouter slug is an auction (eighteen providers behind deepseek-v4-flash, $0.068 to $0.200 input). sort: price + pinned quantization for measurement; live-priced tables before any spend decision. Anchor: docs/agents/model-cost-openrouter.md, including its two recorded corrections.
8. Honesty invariants are harness-owned. Fabrication is caught by checkers and refused by the server, never merely discouraged in prose. The quality bar ("never ship a fabricated result") applies to the MODEL'S output the same way it applies to our code.

Current state (audited 2026-08-16)#

Assets — already built, load-bearing for SMX#

Gap register — what "smallest model, excellent behavior" still needs#

Each gap gets closed by a numbered phase; the TODOS file carries the tasks.

• SMX-G1 — Deck too small to decide anything. 7 golden cases, member-side only, single-shot, no reliability statistics, no committed scorecard, no ratchet. You cannot downshift a model or land a skill against 7 cases.
• SMX-G2 — Monolithic prompt. One conduct block (~20 rules covering member data, audits, tours, highlighting, ids) plus persona, memory, anchors registry, page context, tour catalog, workbench block — assembled for every turn regardless of task (routes/assistant.ts:2117). No task families, no just-in-time loading.
• SMX-G3 — Full tool surface every turn. Up to ~60 tools offered at once in admin scope. Measured harm on the champion model: tara_favorites skipped 4/20 (Phase 4.1 record). No per-task scoping, no forced tool choice.
• SMX-G4 — No uniform argument boundary. Tool args go from the provider straight to per-tool execute; no shared schema validation, no instructive error contract, no single-repair retry. Small models misfill arguments far more often than frontier models; today each tool improvises its response.
• SMX-G5 — Designed-but-unbuilt grounding checks. EVE-VIS-080 (refuse member-owned-data claims when no member-data tool ran THIS turn) is written down as "the real mechanism" in a comment and tracked, not built. No id/title/count citation check exists (EVE-VIS-211 full-id rule is prompt-only plus one store-side resolve).
• SMX-G6 — No escalation ladder. A wrong or checker-refused cheap turn has no verified-failure retry and no bounded model bump; quality problems become member-visible instead of costing one extra iteration.
• SMX-G7 — One binding for every job. The same model serves smalltalk, six-tool audit protocol turns, and workbench mutations; router/judge legs don't exist as bindings. No per-leg registry, no downshift protocol.
• SMX-G8 — Telemetry can't answer SMX questions. AssistantTurnMetricRecord lacks cost, cached-token counts, task family, skill version, checker verdicts, escalation tier, per-tool error names. No failure taxonomy, no cost report.
• SMX-G9 — Zero exemplars anywhere. The prompt is all rules, no worked examples. Few-shot exemplars are the single highest-leverage prompt content for small models and the easiest thing for a strong model to author and verify.
• SMX-G10 — Cache-hostile prompt order. The session-dynamic engine prompt (persona/daypart) is the FIRST prompt part; every stable byte after it is un-cacheable on providers with prefix caching. Order is a pure cost lever at zero behavior change — if the routed provider caches; measured, not assumed.
• SMX-G11 — No model-facing adversarial coverage. Nothing probes prompt injection via tool results (docs snippets, thread bodies, page context carrying instructions), fabrication traps against empty stores, or small-model-specific failure shapes. Small models are more injectable and more suggestible; the deck must measure it.
• SMX-G12 — Battery judging is manual. Ten conversations are captured "for judging" with no automated rubric judge, so multi-turn quality has no regression signal.
• SMX-G13 — No abort propagation (verified 2026-08-16, pi review). runAssistantAgentTurn carries no AbortSignal: a member closing the panel mid-turn does not cancel the provider stream — tokens keep billing and tools keep executing to turn end. pi-ai treats combined abort signals as first-class; Eve should too.
• SMX-G14 — No wire-level turn inspector. Nothing records exactly what the model READ (assembled prompt parts, tools offered, per-iteration payloads), so distillery work starts blind. pi's debug ethos — the exact request is always inspectable — is the standard to meet.

Target architecture#

Twelve components. Wiring points named so the TODOS tasks are checkable. Everything lands above LLMProviderInterface; nothing binds to a vendor.

C1 — Task-family router (deterministic first)#

A pure function resolveTaskFamily(turnFacts) → { family, why } consulted by the turns route before prompt/toolset assembly. Families v1:

member-data · docs · navigate · tour · audit · workbench-read · workbench-write · capability-smalltalk · general (fallback = today's full surface, so routing FAILS OPEN to current behavior).

Priority of signals, highest first — server state beats text:

1. Active audit run in the audit store → audit (state, not classification).
2. Pending confirm-card reply / mutation verbs + admin scope → workbench families.
3. Page context path + client capabilities (tour_player, action_confirm).
4. Keyword/pattern heuristics reusing the deterministic engine's vocabulary and the feature catalog's titles.
5. Nothing matched → general.

A micro-model classifier leg is DEFERRED until misroute telemetry proves the deterministic router insufficient (decision recorded either way). Router is pure, unit-tested, and every routed turn logs {family, why}.

C2 — Skill registry (typed, budgeted, eval-linked)#

apps/oshun/bff/src/assistant/skills/ — one TS module per family exporting a typed AssistantSkill:

interface AssistantSkill {
  id: string; // 'skill.audit.v3'
  family: TaskFamily;
  body: string; // the just-in-time playbook block
  exemplars: readonly SkillExemplar[]; // verified worked examples, ≤2
  toolAllowlist: readonly string[]; // must ⊆ real tool names (spec-enforced)
  checkerIds: readonly string[];
  maxIterations?: number; // audit needs 6; smalltalk needs 2
  evalCaseIds: readonly string[]; // every skill traces to deck cases
}

Registry disciplines, all spec-enforced (not asked politely):

• Token budget: core conduct ≤ ~350 tokens; skill body + exemplars ≤ ~700 (chars/4 heuristic with stated margin — an approximation, documented as one). A budget-busting skill fails the registry spec.
• Traceability lint: a skill with zero evalCaseIds, or naming a deck case that doesn't exist, fails the spec. No unmeasured sentence — the polish initiative's rule, made structural.
• Allowlist integrity: toolAllowlist ⊆ the union of actually-registered tool names, checked against the live builders, so tool renames can't silently orphan a skill.
• Privilege clamp (from the Hermes-parity skill store): at runtime a skill's allowlist INTERSECTS the tools the turn is actually authorized to carry (scope, capabilities) — a skill can only narrow a surface, never widen it (clampAllowedTools semantics, libs/oshun/assistant/src/skills/).
• Versioned ids + a changelog line per edit, because P8's maintenance loop needs to correlate skill versions with telemetry.

The current AGENT_CONDUCT_INSTRUCTIONS block is dismantled INTO this registry: universal honesty/grounding rules stay in the small core; everything family-specific (audit protocol, tour conduct, workbench register, highlight rules) moves into its skill and stops taxing unrelated turns.

C3 — Toolset scoper + forced tool choice#

Per-family filter over the existing buildAssistantAgentToolset output — allowlist projection, nothing rebuilt. Tool-mandatory families (member-data, docs) run with toolChoice: 'required' on the first iteration where the provider supports it (OpenAI-compatible wire does; capability-checked per provider, honest fallback to auto). Target surface per routed turn: ≤ 12 tools instead of ~60. general keeps the full surface — and for it (and any family whose tail of rarely-used tools resists a static allowlist) adopt pi-ai's deferred-tools pattern: tail tools offered as names only plus a load_tools meta-tool that pulls full definitions on demand.

C4 — Constrained argument boundary (validate → instruct → repair once)#

One shared validation layer at the toolset.execute seam: every tool's JSON schema (already declared in its ToolDefinition) is enforced BEFORE the tool body runs. On violation the tool result is a structured instructive error — exact field, expected shape, one inline example — and the runner grants ONE repair iteration for argument errors (bounded; a second violation returns the honest failure to the member). For typed non-tool outputs (router/judge legs, never member prose) use OpenRouter structured outputs (response_format: json_schema) with provider.require_parameters: true so only endpoints that honor the contract serve the call — verified with a positive control the way the quantization flag was (int2 → 400 proves arrival). Where the routed endpoint supports grammar-constrained sampling (pi-ai ships constrained-sampling grammar tool inputs), constrain argument GENERATION itself rather than only validating after — availability through OpenRouter is measured per provider before it is relied on.

C5 — Grounding checkers (the EVE-VIS-080 family, built)#

Pure, unit-tested functions over the turn's observable facts (tool calls + results + reply text), run server-side before turn.complete:

• Member-data grounding (VIS-080): reply claims member-owned data (possessive + domain-noun heuristic, tuned on the deck and honest about being a heuristic) while zero member-data tools ran this turn → hold the reply, inject the verdict, allow one corrective iteration; if it still claims ungrounded data, replace with an honest "let me check" refusal. The 20%→6% prompt sentence stays; the checker is the fix.
• Citation grounding: every full-id-shaped token (work items, threads, tours, decisions) and every catalog title named in the reply must appear in THIS turn's tool results; violations follow the same hold → correct-once → honest-refusal path. Extends EVE-VIS-211 from a request to a mechanism.
• Count sanity: numeric claims about list sizes must match the length the tool returned (exact-extraction cases only; no NLP heroics — stated limit).

Checkers follow the wire-hold pattern already proven for announce-before-act: sentence-atomic streaming is what makes clean interception possible.

C6 — Escalation ladder (verified failure, bounded, logged)#

Per-family policy, default:

1. Tier 1: champion small model (registry-bound).
2. Tier 2: same model, one retry carrying the checker/validator verdict.
3. Tier 3: escalation model — a mid-tier slug chosen in the P7 tournament, cheap-first, NEVER a frontier model by default (CLAUDE.md binding); builder surface may enable a stronger tier-3 by explicit env opt-in.

Every escalation emits {family, tier, reason, caseShape} to telemetry — an escalation is a signed confession that a skill or checker needs work, and P8's maintenance loop consumes exactly that queue. Member-facing latency stays bounded: at most one extra iteration per mechanism, p95 budget per family tracked. The existing per-member token budget stays authoritative above all tiers.

C7 — Per-leg model registry#

apps/oshun/bff/src/assistant/model-registry.ts — the single authority mapping legs → bindings:

Each row: pinned slug, provider preferences (sort: price; fp8 pin for MEASUREMENT arms per A10), price snapshot + date + the command that produced it. Env overrides keep working; the registry is the default, not a cage. Production routing keeps today's semantics unless a row says otherwise.

C8 — Eval deck v2 (the instrument everything else is measured with)#

Grow 7 → 150+ cases, kept in the existing harness (route surface, pure grader). Composition, mapped to sources:

• Ledger-born locks (~50): closed EVE-VIS rows whose failure was MODEL behavior (tool skips, id elision, operator register, envelope leaks, announce-before-act ordering, audit-skip provenance, internal-id leakage) re-expressed as deck cases. Ledger-born cases are non-negotiable: a candidate model or skill that fails one at k=10 is disqualified regardless of aggregate wins — the same "regression lock" semantics the ledger already enforces for code.
• Family coverage (~60): per family × (happy path, empty-store honesty, tool-error honesty, refusal correctness), including the admin/workbench and audit-mode behaviors the current 7 cases don't touch.
• Adversarial (~30): prompt-injection probes through every untrusted channel a tool result can carry (docs snippets, thread bodies, page context, work-item titles containing instructions); fabrication traps (empty favorites, nonexistent docs); id-precision traps; member-vs-operator register traps.
• Multi-turn (~10): the battery's ten conversations, promoted from "captured for judging" into graded multi-turn deck cases (SSE per turn, grader over the full transcript).
• Provider-free self-checks: crisis supersede, budget refusal, confirm bridge refusal — free to run, keep the harness honest.

Grader statistics (new, in-harness): per-case k runs (default k=3; k=10 for promotion decisions); report pass@1 (mean) and pass^k (all k passed — the reliability number that matters for agents); Wilson 95% intervals per family; flake policy per the existing comment — a case flaky on a frontier model poisons regression signal and gets rewritten or dropped, recorded.

Artifacts, committed:

• docs/audits/EVE_SMX_SCORECARD.md — human-readable scorecard per run: per-family pass@1/pass^k + CI, cost per run and median per turn (from usage: {include: true}), latency, model + provider + quantization served, prompt-bytes hash.
• docs/audits/eve-smx-ratchet.json — machine ratchet: per-family floors + the prompt-bytes hash the scorecard was measured against. CI (spending nothing) recomputes the hash of conduct core + skills + registry; a mismatch with no fresh scorecard commit fails the build. You cannot change what the model reads without re-measuring what the model does. Floors only ratchet upward; lowering one requires a human sign-off row in the scorecard, same as the conventions ratchet.

Spend gating unchanged and explicit: live deck runs are operator-invoked only, never CI-automatic; CI runs provider-free cases + grader/scripted-binding specs. Estimated full-deck cost at champion prices: ~600–800 turns ≈ $0.15– 0.40 per k=3 run (estimate, to be replaced by the measured number in the P0 scorecard).

C9 — Rubric judge (validated before trusted)#

For qualities deterministic expectations can't grade (tone, brevity, register, multi-turn coherence): a rubric judge on the judge leg's cheap model. The judge is VALIDATED before it grades anything that gates: ≥40 human-labeled transcripts (drawn from battery + polish evidence), agreement threshold stated in the scorecard; below threshold the judge stays advisory and the case class stays human-judged. Judge model ≠ turn model (self-preference bias), judge prompts live in the skills directory under the same budget/traceability lints.

C10 — The distillery (the strong-model loop, standing)#

The pattern the initiative exists for, made a pipeline rather than a heroic one-off. Claude (or any frontier model, IN SESSION, build-time only):

1. Reads the escalation queue + failure taxonomy + newest deck failures.
2. Writes/edits skills, exemplars, checker rules, tool-description examples.
3. Authors exemplar transcripts by specifying (not roleplaying) ideal tool sequences; exemplars are verified by running the deck, not by looking plausible.
4. Lands everything as ordinary commits measured by deck delta; the ratchet hash forces the re-measure.

Escalation-tier transcripts that PASSED after a tier-2/3 retry are distillation gold: the delta between the failed cheap attempt and the passing retry is exactly what the exemplar should teach. The distillery consumes its own exhaust.

C11 — Telemetry v2 + cost ledger#

Extend AssistantTurnMetricRecord (snapshot-versioned, per its own v1→v2 precedent): costUsd, cacheReadTokens, cacheWriteTokens (pi-ai's Usage splits both cache directions with per-bucket cost — adopt that shape), family, skillVersion, checkerVerdicts, escalationTier, escalationReason, toolErrors (by tool name), iterations. New tools/eve-smx-cost-report.mjs: per-family cost, cache-hit rate, escalation rate, checker-fire rate, top failure shapes — the weekly input to the maintenance loop. Nothing silently rationed: budgets keep their honest-refusal semantics.

C12 — Cache-ordered prompt assembly + context budgets#

Reorder promptParts: stable bytes first (core conduct, then the routed skill), session/turn-dynamic bytes last (persona, daypart, memory recall, page context, anchors — and anchors/tour-catalog blocks only for families whose skills use them). Measure cachedInputTokens before/after on the routed provider; if the auction's cheap endpoint doesn't cache, the reorder still costs nothing and every cache-capable route inherits it. Per-family history window and a server-side digest for long tool results (with an explicit "call again with offset" affordance) complete the context budget. Behavior guard: the deck must be flat within CI across the reorder — this phase buys cost, not behavior.

Two further levers from the pi review: (1) compaction over truncation — today's history is slice(-AGENT_HISTORY_MAX_TURNS), which amputates early context; a rolling summary carried in the dynamic suffix (deterministic or cheap-leg authored, behind a flag, deck flat) preserves it at bounded cost; (2) session cache affinity — pass whatever session/user affinity the routed provider honors (the codex transport already takes sessionId; the champion path passes nothing) and record the finding even when the answer is "none honored".

Model strategy#

• Champion (turn leg): deepseek/deepseek-v4-flash-0731, sort: price, fp8 pinned for measurement arms — the reproducibility trade recorded in A10 stands; do not re-litigate it without re-reading that file.
• Challenger slates are named by CLASS here and priced by ROUTE at decision time (any $ written today would be stale by P7): sub-flash tier for router/judge legs (qwen-flash-class, gemini-flash-lite-class, tiny llama class, :free variants for dev where rate limits allow); flash-class alternates for the turn leg; one mid-tier for the escalation leg.
• Promotion protocol (any leg): price the route live → paired deck runs, same pinned quantization, k=10 → promote only if (a) no ledger-born case newly fails, (b) every family's Wilson lower bound ≥ ratchet floor, (c) cost and p95 latency deltas are stated in the scorecard. Demotion is symmetric and automatic when telemetry floors break (P8).
• Frontier models: never in the serving path, never in tests/harnesses (CLAUDE.md binding); allowed exactly two jobs — the distillery (build-time, in session) and one-off flake audits of deck cases ("would a frontier model pass this deterministically?" — the existing golden-catalog admission rule).

Phase plan (summary — tasks live in the TODOS)#

Phases are strictly ordered — P0 before anything touches a prompt, because an uninstrumented improvement is indistinguishable from a regression with good posture.

Lessons intake — pi / Hermes review (2026-08-16)#

Provenance: read against the installed @earendil-works/pi-ai 0.83.0 source and docs/audits/HERMES_PARITY_TODOS.md after the user asked what agents like pi and Hermes can teach this harness.

Dependency decision — keep pi-ai, pin it exactly. The codex-subscription capability is already ours (our provider, our runner, our auth bridge); pi-ai is only the wire underneath, isolated behind the injectable CodexSubscriptionTransport seam and lazy-loaded. Reimplementing it means owning ~2,500 lines of churn-tracking protocol (codex Responses wire 1,272 + shared Responses conversion/streaming 651 + OAuth refresh 442 + utils), including zstd request bodies, a WebSocket transport variant, session resume, and a rate-limit/billing error taxonomy — against an unofficial backend that changes without notice. Churn absorption is the product the dependency delivers. Hardening instead: the catalog range ^0.83.0 becomes an exact 0.83.0 pin with reviewed bumps (token-touching dep), and the exit ramp is recorded here: if pi-ai ever stalls, implement the seam ourselves — our narrower needs (the CLI owns login; refresh + SSE POST only) size to roughly 600–800 lines plus the churn ownership. Capability noted, not planned: pi-ai also ships Anthropic/Copilot/Kimi subscription OAuth, so a claude-subscription binding would be one thin provider away — explicit-only, operator's call, same honesty caveat as codex.

Adopted lessons and where each lands:

Deliberately not copied: model-writable, runtime-growing prompt skills on the member surface — curated, budgeted, eval-locked skills are the point; and a north star borrowed from pi in one line: nothing in the turn is invisible to the operator, nothing in the prompt is unmeasured.

Risk register#

Success criteria (the initiative's own honesty bar)#

1. Quality: the full deck passes on the pinned champion at floors ≥ the P0 frontier-baseline arm (P0 runs one calibration arm on a stronger model to set "what good looks like" — a measurement, not a serving-path binding), and every ledger-born lock holds at k=10.
2. Cost: median member turn cost and cache-hit rate published in the scorecard, trending down across P1/P2 with the deltas attributed to named mechanisms, not vibes.
3. Downshift: every leg bound to the cheapest model that passes the promotion protocol, with the decision table and live-priced routes committed.
4. Sustainment: the maintenance loop runs on telemetry, not enthusiasm — escalation queue → distillery → deck delta — and the ratchet makes silent decay structurally impossible.

Appendix A — deck case sources map (P0 mining guide)#

Appendix B — glossary#

• pass@1 — mean pass rate over k independent runs of a case.
• pass^k — fraction of cases passing ALL k runs; the reliability number.
• Ledger-born lock — a deck case derived from a closed EVE-VIS row; failing one disqualifies a change outright.
• Leg — an independently model-bound job (turn, escalation, judge, router, embedding).
• Distillery — the build-time strong-model loop that turns failures into skills, exemplars, and checkers.
• Ratchet — committed per-family floors + prompt-bytes hash; CI fails on prompt drift without a fresh scorecard.