# Eve Small-Model Excellence (SMX) — Design (2026-08-16) Companion execution ledger: `EVE_SMALL_MODEL_EXCELLENCE_TODOS_2026-08-16.md` (root). Status: DESIGN — nothing below is shipped unless a TODOS checkbox says so. ## Thesis Run the assistant brain (member-facing Lilith + builder-facing Eve, one runner) on the smallest model that passes a regression-locked evaluation deck — today's champion is `deepseek/deepseek-v4-flash-0731` at ~$0.10/$0.18 per 1M tokens, ~100x under frontier — and make that sustainable by moving intelligence out of the model and into the harness: a strong model (Claude, at build time, in session) does the capability analysis, writes the skills, designs the checkers, and distills the exemplars; the small model executes inside a harness that routes, scopes, validates, verifies, and escalates. The deck is the contract between the two: no skill line, prompt edit, checker, or model swap lands without a measured delta. This is not a new philosophy for this repo — it is the Eve polish initiative's own discovery, systematized. The polish work already proved (with 20-run arms and Fisher exact tests) that a prompt sentence is a request and a server-side check is a mechanism (`routes/assistant.ts:1143` block, EVE-VIS-128/277, the announce-before-act wire hold). SMX finishes the thought: **compile-time intelligence, run-time economy.** ## Scope **In scope:** the assistant brain end to end — provider bindings (`apps/oshun/bff/src/assistant/agent-provider-config.ts`), the turn runner (`agent-turn-runner.ts`), prompt assembly and toolset assembly (`routes/assistant.ts` streaming turn route), all tool surfaces (member domain adapters, docs, tours, audit, admin, workbench read/write), the eval harness (`evals/`), turn metrics, and the OpenRouter routing configuration in `@oshun/ai`. **Out of scope:** the codex coding-agent lane (`tools/eve-codex-agent.mjs`, subscription-billed, its own loop); the deterministic engine's message route (already model-free); voice STT/TTS vendors; member-web panel rendering (polish initiative owns it). The pi-ai subscription transport (`CodexSubscriptionProvider`) stays as an explicit-only provider and benefits from every harness change for free, since all of SMX lands above the `LLMProviderInterface` seam. **Adjacent, not forked:** the repo has a second brain — `@oshun/assistant`, the Hermes-parity autonomous agent (channels, scheduling, its own Voyager-style skill store; `docs/audits/HERMES_PARITY_TODOS.md`). SMX builds for the BFF brain, but the deck harness pattern, skill disciplines, and checkers are designed so that stack can consume them too — two brains, one measurement doctrine. ## Doctrine Eight principles. Each is anchored to evidence already in this repo; none is aspiration. 1. **Mechanism over request.** A rule the model must obey becomes a server-side check, not (only) a prompt sentence. Anchor: EVE-VIS-128 (audit-skip provenance enforced by the store), EVE-VIS-277 (mutating tools refuse without the confirm bridge), the announce-before-act stream hold, and the measured verdict "this sentence is NOT a fix" (20%→6%, p=0.34) in `routes/assistant.ts:1143`. 2. **Evals decide, not vibes.** Every prompt, skill, checker, and model change is judged by the deck, with pinned routing and stated statistics. Anchor: the repo's Fisher-exact discipline and the existing pure, unit-tested grader ("a vacuous grader cannot hide", `evals/eval-harness.ts`). 3. **Compile-time intelligence, run-time economy.** The strong model is a compiler: it analyzes failures, writes skills, authors exemplars, proposes checkers — at build time, in session, costing nothing at runtime. The small model is the runtime. The strong model NEVER sits in the serving path (CLAUDE.md cost rule). 4. **Smallest model per LEG, not per product.** Turn execution, routing, judging, and embedding are separate legs with separate bindings; each leg gets the cheapest model that passes its own gate. A product-wide "we use model X" is a category error. 5. **Context is a budget, and small models pay it first.** Long undifferentiated prompts are where flash-class models drown: today every tool-capable turn carries the full conduct block, the anchor registry, and the curated tour catalog regardless of task, and an admin turn can offer ~60 tools. Anchor: the measured `tara_favorites` skip (4/20) under the full surface. 6. **Every failure becomes a case; every fix becomes a mechanism, a skill line, or an exemplar — traceable both ways.** Anchor: the defect ledger (279 rows, every closed S1/S2 with a runnable lock, `tools/eve-polish/ ledger-lock-audit.mjs`). SMX extends the same discipline to model behavior. 7. **Price the route, not the model.** An OpenRouter slug is an auction (eighteen providers behind `deepseek-v4-flash`, $0.068 to $0.200 input). `sort: price` + pinned quantization for measurement; live-priced tables before any spend decision. Anchor: `docs/agents/model-cost-openrouter.md`, including its two recorded corrections. 8. **Honesty invariants are harness-owned.** Fabrication is caught by checkers and refused by the server, never merely discouraged in prose. The quality bar ("never ship a fabricated result") applies to the MODEL'S output the same way it applies to our code. ## Current state (audited 2026-08-16) ### Assets — already built, load-bearing for SMX | id | asset | where | | --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | A1 | Bounded provider-agnostic turn loop; sentence-atomic streaming; scaffolding-tail drop; envelope unwrap; tools-off final iteration; honest `refused` outcome | `agent-turn-runner.ts` | | A2 | Five provider bindings behind one interface (anthropic, openai, openrouter, codex-cli, codex-subscription); fail-closed resolution | `agent-provider-config.ts`, `@oshun/ai` | | A3 | ~38 member tools + admin/audit + ~17 workbench tools, server-injected auth, typed definitions | `agent-tools.ts`, `admin-agent-tools.ts`, `audit-agent-tools.ts` | | A4 | Confirm bridge; `mutating` tools refuse without it; actor attribution per conversation+turn | `action-confirm.ts`, workbench bindings | | A5 | Server-side behavior checkers on the wire: safety supersede (stream + completion), announce-before-act hold, audit-skip provenance, completion checker | `routes/assistant.ts` | | A6 | Golden eval harness at the REAL route surface with a pure unit-tested grader; live runs credential-gated (`pnpm --filter @oshun/bff eval:assistant`) | `evals/eval-harness.ts`, `evals/assistant-golden.eval.ts` | | A7 | Ten-conversation live battery, transcript capture | `tools/eve-conversation-battery.mjs` | | A8 | Defect ledger: 279 rows, zero open, machine-audited locks | `docs/audits/EVE_POLISH_DEFECT_LEDGER_2026-08.md` | | A9 | Durable turn metrics + per-member/tenant token budgets with honest refusal | `turn-metrics.ts` | | A10 | Cost doctrine: price-the-route method, `OPENROUTER_PROVIDER_SORT=price`, quantization pinning, the `-0731` reproducibility decision | `docs/agents/model-cost-openrouter.md` | | A11 | Deterministic engine fallback (agent unconfigured → honest `agent_not_configured`) | engine + routes | | A12 | Feature catalog (typed graph, CI integrity gates) + durable audit store | `@oshun/shell-assistant/feature-catalog`, `audit-store.ts` | | A13 | Docs search over the curated corpus (member) / full estate (admin) | `docs-search.ts` | ### Gap register — what "smallest model, excellent behavior" still needs Each gap gets closed by a numbered phase; the TODOS file carries the tasks. - **SMX-G1 — Deck too small to decide anything.** 7 golden cases, member-side only, single-shot, no reliability statistics, no committed scorecard, no ratchet. You cannot downshift a model or land a skill against 7 cases. - **SMX-G2 — Monolithic prompt.** One conduct block (~20 rules covering member data, audits, tours, highlighting, ids) plus persona, memory, anchors registry, page context, tour catalog, workbench block — assembled for every turn regardless of task (`routes/assistant.ts:2117`). No task families, no just-in-time loading. - **SMX-G3 — Full tool surface every turn.** Up to ~60 tools offered at once in admin scope. Measured harm on the champion model: `tara_favorites` skipped 4/20 (Phase 4.1 record). No per-task scoping, no forced tool choice. - **SMX-G4 — No uniform argument boundary.** Tool args go from the provider straight to per-tool `execute`; no shared schema validation, no instructive error contract, no single-repair retry. Small models misfill arguments far more often than frontier models; today each tool improvises its response. - **SMX-G5 — Designed-but-unbuilt grounding checks.** EVE-VIS-080 (refuse member-owned-data claims when no member-data tool ran THIS turn) is written down as "the real mechanism" in a comment and tracked, not built. No id/title/count citation check exists (EVE-VIS-211 full-id rule is prompt-only plus one store-side resolve). - **SMX-G6 — No escalation ladder.** A wrong or checker-refused cheap turn has no verified-failure retry and no bounded model bump; quality problems become member-visible instead of costing one extra iteration. - **SMX-G7 — One binding for every job.** The same model serves smalltalk, six-tool audit protocol turns, and workbench mutations; router/judge legs don't exist as bindings. No per-leg registry, no downshift protocol. - **SMX-G8 — Telemetry can't answer SMX questions.** `AssistantTurnMetricRecord` lacks cost, cached-token counts, task family, skill version, checker verdicts, escalation tier, per-tool error names. No failure taxonomy, no cost report. - **SMX-G9 — Zero exemplars anywhere.** The prompt is all rules, no worked examples. Few-shot exemplars are the single highest-leverage prompt content for small models and the easiest thing for a strong model to author and verify. - **SMX-G10 — Cache-hostile prompt order.** The session-dynamic engine prompt (persona/daypart) is the FIRST prompt part; every stable byte after it is un-cacheable on providers with prefix caching. Order is a pure cost lever at zero behavior change — if the routed provider caches; measured, not assumed. - **SMX-G11 — No model-facing adversarial coverage.** Nothing probes prompt injection via tool results (docs snippets, thread bodies, page context carrying instructions), fabrication traps against empty stores, or small-model-specific failure shapes. Small models are more injectable and more suggestible; the deck must measure it. - **SMX-G12 — Battery judging is manual.** Ten conversations are captured "for judging" with no automated rubric judge, so multi-turn quality has no regression signal. - **SMX-G13 — No abort propagation (verified 2026-08-16, pi review).** `runAssistantAgentTurn` carries no `AbortSignal`: a member closing the panel mid-turn does not cancel the provider stream — tokens keep billing and tools keep executing to turn end. pi-ai treats combined abort signals as first-class; Eve should too. - **SMX-G14 — No wire-level turn inspector.** Nothing records exactly what the model READ (assembled prompt parts, tools offered, per-iteration payloads), so distillery work starts blind. pi's debug ethos — the exact request is always inspectable — is the standard to meet. ## Target architecture Twelve components. Wiring points named so the TODOS tasks are checkable. Everything lands above `LLMProviderInterface`; nothing binds to a vendor. ### C1 — Task-family router (deterministic first) A pure function `resolveTaskFamily(turnFacts) → { family, why }` consulted by the turns route before prompt/toolset assembly. Families v1: `member-data` · `docs` · `navigate` · `tour` · `audit` · `workbench-read` · `workbench-write` · `capability-smalltalk` · `general` (fallback = today's full surface, so routing FAILS OPEN to current behavior). Priority of signals, highest first — **server state beats text**: 1. Active audit run in the audit store → `audit` (state, not classification). 2. Pending confirm-card reply / mutation verbs + admin scope → workbench families. 3. Page context path + client capabilities (tour_player, action_confirm). 4. Keyword/pattern heuristics reusing the deterministic engine's vocabulary and the feature catalog's titles. 5. Nothing matched → `general`. A micro-model classifier leg is DEFERRED until misroute telemetry proves the deterministic router insufficient (decision recorded either way). Router is pure, unit-tested, and every routed turn logs `{family, why}`. ### C2 — Skill registry (typed, budgeted, eval-linked) `apps/oshun/bff/src/assistant/skills/` — one TS module per family exporting a typed `AssistantSkill`: ```ts interface AssistantSkill { id: string; // 'skill.audit.v3' family: TaskFamily; body: string; // the just-in-time playbook block exemplars: readonly SkillExemplar[]; // verified worked examples, ≤2 toolAllowlist: readonly string[]; // must ⊆ real tool names (spec-enforced) checkerIds: readonly string[]; maxIterations?: number; // audit needs 6; smalltalk needs 2 evalCaseIds: readonly string[]; // every skill traces to deck cases } ``` Registry disciplines, all spec-enforced (not asked politely): - **Token budget:** core conduct ≤ ~350 tokens; skill body + exemplars ≤ ~700 (chars/4 heuristic with stated margin — an approximation, documented as one). A budget-busting skill fails the registry spec. - **Traceability lint:** a skill with zero `evalCaseIds`, or naming a deck case that doesn't exist, fails the spec. No unmeasured sentence — the polish initiative's rule, made structural. - **Allowlist integrity:** `toolAllowlist ⊆` the union of actually-registered tool names, checked against the live builders, so tool renames can't silently orphan a skill. - **Privilege clamp (from the Hermes-parity skill store):** at runtime a skill's allowlist INTERSECTS the tools the turn is actually authorized to carry (scope, capabilities) — a skill can only narrow a surface, never widen it (`clampAllowedTools` semantics, `libs/oshun/assistant/src/skills/`). - Versioned ids + a changelog line per edit, because P8's maintenance loop needs to correlate skill versions with telemetry. The current `AGENT_CONDUCT_INSTRUCTIONS` block is dismantled INTO this registry: universal honesty/grounding rules stay in the small core; everything family-specific (audit protocol, tour conduct, workbench register, highlight rules) moves into its skill and stops taxing unrelated turns. ### C3 — Toolset scoper + forced tool choice Per-family filter over the existing `buildAssistantAgentToolset` output — allowlist projection, nothing rebuilt. Tool-mandatory families (`member-data`, `docs`) run with `toolChoice: 'required'` on the first iteration where the provider supports it (OpenAI-compatible wire does; capability-checked per provider, honest fallback to `auto`). Target surface per routed turn: ≤ 12 tools instead of ~60. `general` keeps the full surface — and for it (and any family whose tail of rarely-used tools resists a static allowlist) adopt pi-ai's deferred-tools pattern: tail tools offered as names only plus a `load_tools` meta-tool that pulls full definitions on demand. ### C4 — Constrained argument boundary (validate → instruct → repair once) One shared validation layer at the `toolset.execute` seam: every tool's JSON schema (already declared in its `ToolDefinition`) is enforced BEFORE the tool body runs. On violation the tool result is a structured instructive error — exact field, expected shape, one inline example — and the runner grants ONE repair iteration for argument errors (bounded; a second violation returns the honest failure to the member). For typed non-tool outputs (router/judge legs, never member prose) use OpenRouter structured outputs (`response_format: json_schema`) with `provider.require_parameters: true` so only endpoints that honor the contract serve the call — verified with a positive control the way the quantization flag was (`int2` → 400 proves arrival). Where the routed endpoint supports grammar-constrained sampling (pi-ai ships constrained-sampling grammar tool inputs), constrain argument GENERATION itself rather than only validating after — availability through OpenRouter is measured per provider before it is relied on. ### C5 — Grounding checkers (the EVE-VIS-080 family, built) Pure, unit-tested functions over the turn's observable facts (tool calls + results + reply text), run server-side before `turn.complete`: - **Member-data grounding (VIS-080):** reply claims member-owned data (possessive + domain-noun heuristic, tuned on the deck and honest about being a heuristic) while zero member-data tools ran this turn → hold the reply, inject the verdict, allow one corrective iteration; if it still claims ungrounded data, replace with an honest "let me check" refusal. The 20%→6% prompt sentence stays; the checker is the fix. - **Citation grounding:** every full-id-shaped token (work items, threads, tours, decisions) and every catalog title named in the reply must appear in THIS turn's tool results; violations follow the same hold → correct-once → honest-refusal path. Extends EVE-VIS-211 from a request to a mechanism. - **Count sanity:** numeric claims about list sizes must match the length the tool returned (exact-extraction cases only; no NLP heroics — stated limit). Checkers follow the wire-hold pattern already proven for announce-before-act: sentence-atomic streaming is what makes clean interception possible. ### C6 — Escalation ladder (verified failure, bounded, logged) Per-family policy, default: 1. **Tier 1:** champion small model (registry-bound). 2. **Tier 2:** same model, one retry carrying the checker/validator verdict. 3. **Tier 3:** escalation model — a mid-tier slug chosen in the P7 tournament, cheap-first, NEVER a frontier model by default (CLAUDE.md binding); builder surface may enable a stronger tier-3 by explicit env opt-in. Every escalation emits `{family, tier, reason, caseShape}` to telemetry — an escalation is a signed confession that a skill or checker needs work, and P8's maintenance loop consumes exactly that queue. Member-facing latency stays bounded: at most one extra iteration per mechanism, p95 budget per family tracked. The existing per-member token budget stays authoritative above all tiers. ### C7 — Per-leg model registry `apps/oshun/bff/src/assistant/model-registry.ts` — the single authority mapping legs → bindings: | leg | job | default binding | | ---------- | ---------------------------------- | --------------------------------------- | | turn | member/builder turns (per family) | `deepseek/deepseek-v4-flash-0731` (pin) | | escalation | tier-3 retries | chosen in P7, mid-tier | | judge | deck rubric judging (build/eval) | chosen in P7, must ≠ turn leg model | | router | residual classification (deferred) | none until misroute data demands it | | embedding | docs search (exists) | unchanged | Each row: pinned slug, provider preferences (`sort: price`; fp8 pin for MEASUREMENT arms per A10), price snapshot + date + the command that produced it. Env overrides keep working; the registry is the default, not a cage. Production routing keeps today's semantics unless a row says otherwise. ### C8 — Eval deck v2 (the instrument everything else is measured with) Grow 7 → 150+ cases, kept in the existing harness (route surface, pure grader). Composition, mapped to sources: - **Ledger-born locks (~50):** closed EVE-VIS rows whose failure was MODEL behavior (tool skips, id elision, operator register, envelope leaks, announce-before-act ordering, audit-skip provenance, internal-id leakage) re-expressed as deck cases. **Ledger-born cases are non-negotiable:** a candidate model or skill that fails one at k=10 is disqualified regardless of aggregate wins — the same "regression lock" semantics the ledger already enforces for code. - **Family coverage (~60):** per family × (happy path, empty-store honesty, tool-error honesty, refusal correctness), including the admin/workbench and audit-mode behaviors the current 7 cases don't touch. - **Adversarial (~30):** prompt-injection probes through every untrusted channel a tool result can carry (docs snippets, thread bodies, page context, work-item titles containing instructions); fabrication traps (empty favorites, nonexistent docs); id-precision traps; member-vs-operator register traps. - **Multi-turn (~10):** the battery's ten conversations, promoted from "captured for judging" into graded multi-turn deck cases (SSE per turn, grader over the full transcript). - **Provider-free self-checks:** crisis supersede, budget refusal, confirm bridge refusal — free to run, keep the harness honest. Grader statistics (new, in-harness): per-case k runs (default k=3; k=10 for promotion decisions); report **pass@1** (mean) and **pass^k** (all k passed — the reliability number that matters for agents); Wilson 95% intervals per family; flake policy per the existing comment — a case flaky on a frontier model poisons regression signal and gets rewritten or dropped, recorded. Artifacts, committed: - `docs/audits/EVE_SMX_SCORECARD.md` — human-readable scorecard per run: per-family pass@1/pass^k + CI, cost per run and median per turn (from `usage: {include: true}`), latency, model + provider + quantization served, prompt-bytes hash. - `docs/audits/eve-smx-ratchet.json` — machine ratchet: per-family floors + the prompt-bytes hash the scorecard was measured against. CI (spending nothing) recomputes the hash of conduct core + skills + registry; a mismatch with no fresh scorecard commit fails the build. **You cannot change what the model reads without re-measuring what the model does.** Floors only ratchet upward; lowering one requires a human sign-off row in the scorecard, same as the conventions ratchet. Spend gating unchanged and explicit: live deck runs are operator-invoked only, never CI-automatic; CI runs provider-free cases + grader/scripted-binding specs. Estimated full-deck cost at champion prices: ~600–800 turns ≈ $0.15– 0.40 per k=3 run (estimate, to be replaced by the measured number in the P0 scorecard). ### C9 — Rubric judge (validated before trusted) For qualities deterministic expectations can't grade (tone, brevity, register, multi-turn coherence): a rubric judge on the judge leg's cheap model. The judge is VALIDATED before it grades anything that gates: ≥40 human-labeled transcripts (drawn from battery + polish evidence), agreement threshold stated in the scorecard; below threshold the judge stays advisory and the case class stays human-judged. Judge model ≠ turn model (self-preference bias), judge prompts live in the skills directory under the same budget/traceability lints. ### C10 — The distillery (the strong-model loop, standing) The pattern the initiative exists for, made a pipeline rather than a heroic one-off. Claude (or any frontier model, IN SESSION, build-time only): 1. Reads the escalation queue + failure taxonomy + newest deck failures. 2. Writes/edits skills, exemplars, checker rules, tool-description examples. 3. Authors exemplar transcripts by specifying (not roleplaying) ideal tool sequences; exemplars are verified by running the deck, not by looking plausible. 4. Lands everything as ordinary commits measured by deck delta; the ratchet hash forces the re-measure. Escalation-tier transcripts that PASSED after a tier-2/3 retry are distillation gold: the delta between the failed cheap attempt and the passing retry is exactly what the exemplar should teach. The distillery consumes its own exhaust. ### C11 — Telemetry v2 + cost ledger Extend `AssistantTurnMetricRecord` (snapshot-versioned, per its own v1→v2 precedent): `costUsd`, `cacheReadTokens`, `cacheWriteTokens` (pi-ai's `Usage` splits both cache directions with per-bucket cost — adopt that shape), `family`, `skillVersion`, `checkerVerdicts`, `escalationTier`, `escalationReason`, `toolErrors` (by tool name), `iterations`. New `tools/eve-smx-cost-report.mjs`: per-family cost, cache-hit rate, escalation rate, checker-fire rate, top failure shapes — the weekly input to the maintenance loop. Nothing silently rationed: budgets keep their honest-refusal semantics. ### C12 — Cache-ordered prompt assembly + context budgets Reorder `promptParts`: stable bytes first (core conduct, then the routed skill), session/turn-dynamic bytes last (persona, daypart, memory recall, page context, anchors — and anchors/tour-catalog blocks only for families whose skills use them). Measure `cachedInputTokens` before/after on the routed provider; if the auction's cheap endpoint doesn't cache, the reorder still costs nothing and every cache-capable route inherits it. Per-family history window and a server-side digest for long tool results (with an explicit "call again with offset" affordance) complete the context budget. Behavior guard: the deck must be flat within CI across the reorder — this phase buys cost, not behavior. Two further levers from the pi review: (1) **compaction over truncation** — today's history is `slice(-AGENT_HISTORY_MAX_TURNS)`, which amputates early context; a rolling summary carried in the dynamic suffix (deterministic or cheap-leg authored, behind a flag, deck flat) preserves it at bounded cost; (2) **session cache affinity** — pass whatever session/user affinity the routed provider honors (the codex transport already takes `sessionId`; the champion path passes nothing) and record the finding even when the answer is "none honored". ## Model strategy - **Champion (turn leg):** `deepseek/deepseek-v4-flash-0731`, `sort: price`, fp8 pinned for measurement arms — the reproducibility trade recorded in A10 stands; do not re-litigate it without re-reading that file. - **Challenger slates** are named by CLASS here and priced by ROUTE at decision time (any $ written today would be stale by P7): sub-flash tier for router/judge legs (qwen-flash-class, gemini-flash-lite-class, tiny llama class, `:free` variants for dev where rate limits allow); flash-class alternates for the turn leg; one mid-tier for the escalation leg. - **Promotion protocol (any leg):** price the route live → paired deck runs, same pinned quantization, k=10 → promote only if (a) no ledger-born case newly fails, (b) every family's Wilson lower bound ≥ ratchet floor, (c) cost and p95 latency deltas are stated in the scorecard. Demotion is symmetric and automatic when telemetry floors break (P8). - **Frontier models:** never in the serving path, never in tests/harnesses (CLAUDE.md binding); allowed exactly two jobs — the distillery (build-time, in session) and one-off flake audits of deck cases ("would a frontier model pass this deterministically?" — the existing golden-catalog admission rule). ## Phase plan (summary — tasks live in the TODOS) | phase | name | closes | exit gate | | ----- | ---------------------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | P0 | Instrument & baseline | G1, G13, G14, G8 (part) | ≥120 graded cases across all families incl. adversarial; telemetry v2 fields land; abort propagation + turn-trace capture landed; baseline scorecard + ratchet committed; ZERO prompt/tool changes in P0 | | P1 | Cache order & context budgets | G10 | deck flat within CI; cached-token + cost delta measured and recorded; compaction flag + affinity findings recorded | | P2 | Router, toolset scoping, skills v1 | G2, G3, G9 (part) | tool-selection families strictly better; no family below floor; prompt tokens/turn and tools/turn reduced, measured; misroute rate < stated threshold | | P3 | Constrained argument boundary | G4 | tool-arg error rate down, measured; repair loop bounded; no floor broken | | P4 | Grounding checkers | G5 | adversarial fabrication family improves; VIS-080 shape ~0 at k=10; wire holds verified at the SSE level | | P5 | Escalation ladder + model registry | G6, G7 | effective pass^k with ladder vs cost curve published; escalation rate < threshold; zero unlogged escalations | | P6 | Distillery pass 1 (exemplars) | G9 | measured family deltas from exemplars alone; skills changelog + traceability complete | | P7 | Downshift tournament | G7 (finish) | every leg on the cheapest model passing protocol; decision table with live-priced routes committed | | P8 | Sustainment loop | G12 + drift | maintenance runbook live; held-out case set reporting separately; monthly route-price note; demotion triggers wired | Phases are strictly ordered — P0 before anything touches a prompt, because an uninstrumented improvement is indistinguishable from a regression with good posture. ## Lessons intake — pi / Hermes review (2026-08-16) Provenance: read against the installed `@earendil-works/pi-ai` 0.83.0 source and `docs/audits/HERMES_PARITY_TODOS.md` after the user asked what agents like pi and Hermes can teach this harness. **Dependency decision — keep pi-ai, pin it exactly.** The codex-subscription capability is already ours (our provider, our runner, our auth bridge); pi-ai is only the wire underneath, isolated behind the injectable `CodexSubscriptionTransport` seam and lazy-loaded. Reimplementing it means owning ~2,500 lines of churn-tracking protocol (codex Responses wire 1,272 + shared Responses conversion/streaming 651 + OAuth refresh 442 + utils), including zstd request bodies, a WebSocket transport variant, session resume, and a rate-limit/billing error taxonomy — against an unofficial backend that changes without notice. Churn absorption is the product the dependency delivers. Hardening instead: the catalog range `^0.83.0` becomes an exact `0.83.0` pin with reviewed bumps (token-touching dep), and the exit ramp is recorded here: if pi-ai ever stalls, implement the seam ourselves — our narrower needs (the CLI owns login; refresh + SSE POST only) size to roughly 600–800 lines plus the churn ownership. Capability noted, not planned: pi-ai also ships Anthropic/Copilot/Kimi subscription OAuth, so a `claude-subscription` binding would be one thin provider away — explicit-only, operator's call, same honesty caveat as codex. Adopted lessons and where each lands: | lesson | source | lands | | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | --------------- | | Abort propagation route→runner→provider | pi-ai `combineAbortSignals` (verified absent in Eve) | G13 → P0 | | Wire-level turn-trace capture (what the model READ) | pi's inspect-the-exact-request ethos | G14 → P0 | | Cache read/write token + cost split in telemetry | pi-ai `Usage` shape | C11 → P0 (0.14) | | History compaction over truncation | pi session management | C12 → P1 | | Session cache affinity on the champion path | pi-ai prompt-cache keys; our codex `sessionId` | C12 → P1 | | Deferred tools (names only + `load_tools` meta-tool) | pi-ai `splitDeferredTools` | C3 → P2 | | Privilege clamp: skill allowlist ∩ turn scope, never widen | `@oshun/assistant` `clampAllowedTools` (Hermes P3) | C2 → P2 | | Grammar-constrained argument sampling where endpoints support it | pi-ai constrained-sampling | C4 → P3 | | Retrieval-ranked skill hints for `general`; model-authored skills feed the distillery queue, never the prompt directly | Hermes Voyager-style skill store | C2/C10 → P2/P6 | Deliberately not copied: model-writable, runtime-growing prompt skills on the member surface — curated, budgeted, eval-locked skills are the point; and a north star borrowed from pi in one line: **nothing in the turn is invisible to the operator, nothing in the prompt is unmeasured.** ## Risk register | risk | mitigation | | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Goodhart / deck overfitting** — skills tuned to cases, not behavior | Held-out set authored AFTER skills exist, never tuned against, reported separately (P8); adversarial cases refreshed per phase; ledger-born locks immune to aggregate gaming | | **Router misroutes** — wrong family = wrong tools = confident nonsense | Fail-open to `general` (today's behavior); `{family, why}` logged every turn; misroute taxonomy reviewed at P2 exit before scoping tightens further | | **Engine-prompt split is the riskiest refactor** (session prompt is dynamic-first) | Behind a flag; byte-diff spec proves old order reproducible; deck flat within CI required to keep the flag on | | **Provider auction drift** — the cheap endpoint changes under the pin | Monthly price-the-route note in the scorecard (A10 command); `sort: price` stability finding already recorded; quantization pinned for every measurement arm | | **Latency from repair/escalation loops** | Hard bound: one extra iteration per mechanism; per-family p95 in telemetry; smalltalk family capped at 2 iterations | | **Skill/token budget creep** — the prompt regrows one "important" sentence at a time | Budget is a failing spec, not a guideline; traceability lint makes every sentence carry a case id | | **Judge circularity / self-preference** | Judge validated on human labels before gating; judge model ≠ turn model; below-threshold judges stay advisory | | **Small-model injectability** — tool results carrying instructions | Measured by the adversarial family; capability scoping is the real defense (mutating tools already refuse without the bridge; scoped toolsets shrink the blast radius); checkers catch ungrounded claims | | **Eval spend surprises** | Live runs operator-invoked only; cost printed per run in the scorecard; k=10 reserved for promotions | | **This Mac** (16 GB, three freezes on record) | Deck runs are network-bound: BFF + Postgres only, no Next dev server needed; memory checkpoint before every run; PIDs tracked and killed per CLAUDE.md | ## Success criteria (the initiative's own honesty bar) 1. **Quality:** the full deck passes on the pinned champion at floors ≥ the P0 frontier-baseline arm (P0 runs one calibration arm on a stronger model to set "what good looks like" — a measurement, not a serving-path binding), and every ledger-born lock holds at k=10. 2. **Cost:** median member turn cost and cache-hit rate published in the scorecard, trending down across P1/P2 with the deltas attributed to named mechanisms, not vibes. 3. **Downshift:** every leg bound to the cheapest model that passes the promotion protocol, with the decision table and live-priced routes committed. 4. **Sustainment:** the maintenance loop runs on telemetry, not enthusiasm — escalation queue → distillery → deck delta — and the ratchet makes silent decay structurally impossible. ## Appendix A — deck case sources map (P0 mining guide) | source | case class | | ---------------------------------------------------- | ------------------------------------------------------------------------------------- | | EVE-VIS-080 record (4/20 tool skip) | member-data grounding traps, k=10 | | EVE-VIS-211 (elided ids) | full-id citation cases (workbench) | | EVE-VIS-212 (member experiences offered to operator) | register traps (admin scope) | | EVE-VIS-017 (JSON envelope leak) | envelope/prose cases | | Announce-before-act hold (spine run 12) | SSE-ORDER case: no completed-mark delta before `audit_mark` result ok | | EVE-VIS-128 (skip provenance) | audit-skip refusal + reason-elicitation cases | | Internal-id leakage rule | member-surface id-hygiene cases | | Battery conversations (10) | multi-turn graded cases | | Docs honest-absence (existing) | expanded per corpus tier (member vs admin) | | Crisis/budget/bridge (provider-free) | harness self-checks, run always | | NEW: injection probes | instructions embedded in docs snippets, thread bodies, page context, work-item titles | | NEW: empty-store honesty | zero favorites / zero threads / zero open work — honest empties, no invention | ## Appendix B — glossary - **pass@1** — mean pass rate over k independent runs of a case. - **pass^k** — fraction of cases passing ALL k runs; the reliability number. - **Ledger-born lock** — a deck case derived from a closed EVE-VIS row; failing one disqualifies a change outright. - **Leg** — an independently model-bound job (turn, escalation, judge, router, embedding). - **Distillery** — the build-time strong-model loop that turns failures into skills, exemplars, and checkers. - **Ratchet** — committed per-family floors + prompt-bytes hash; CI fails on prompt drift without a fresh scorecard.