Builder-family eval environment notes (EVE_EVERYWHERE 0.4)

Why the docs and workbench families are "environment-blocked" in the eval (task-families.ts's standing comment), what each blocked case actually needs, and what happened when the environment was supplied locally on 2026-08-19 (k=1, champion deepseek/deepseek-v4-flash-0731, dev Postgres, EVE_SMX_EVAL_CASE_IDS selection; the harness itself prints the honest caveat — "[PARTIAL RUN] … floors and scorecards need the FULL deck" — so nothing below is a floor, only an unblocking probe).

The blocking mechanism (all structural, none in the harness)#

The eval boots the real app (createApp) and inherits process env. The harness already forwards everything the cases declare — operator authScopes, clientCapabilities: ['action_confirm'] — so the block is never machinery, always a missing runtime surface:

1. Workbench tools: the route offers them only when OSHUN_ADMIN_DATABASE_URL ?? OSHUN_V1_DATABASE_URL yields a live store ("no DB, no workbench tools" — routes/assistant.ts). No eval file sets a DB URL, and CI has no seeded Postgres.
2. Docs corpus: search_docs is offered only when a corpus loads with ≥1 chunk. The MEMBER corpus is empty BY DESIGN (EVE-VIS-126 — a docs tool with zero excerpts invites confident fabrication, so it is not offered). The FULL corpus (generated/docs-search-corpus.full.jsonl, 54k lines here) is a machine-local build artifact and is NOT git-tracked — absent on any fresh checkout, whereupon full falls back to the empty member file and the tool vanishes for operators too.

Per-case dependencies and the observed local run#

Run artifacts: /tmp logs from the 2026-08-19 session; verdict lines quoted in the EVE_EVERYWHERE ledger entry for 0.4. Side observation, recorded for accuracy: the original eval:assistant script selected every .eval.ts file. With EVE_SMX_EVAL_CASE_IDS set, battery-transcripts.eval.ts failed loudly ("names X — not a battery case"); without a selection it also captured and overwrote the human-labeling corpus as an unintended side effect of a deck run. The completion re-audit split the two entry points so each operator command selects exactly one live suite.

What Phase 2.1 therefore builds#

• A seeded-Postgres fixture leg for eval runs: OSHUN_V1_DATABASE_URL pointing at a disposable schema seeded with the minimum the cases name (one adr- decision, one open task) — NOT the developer's live dev DB, so eval runs cannot eat local state.
• A corpus step: build (or check in a frozen slice of) the full docs corpus for eval environments, sized for the two docs cases.
• The family-docs-member-scope re-scope decision goes to Phase 2.2 with the EVE-VIS-126 rationale attached; until then the case cannot gate.
• k≥10 with Wilson bounds before any floor claims — the two flaky observations above are exactly why (the SMX k=3 lottery lesson, re-demonstrated at k=1).

Chunk-runner hazards measured 2026-08-20 (the 2.4 foreground pool)#

Three operational truths the 2.4 chunk runs proved, recorded here so the next runner does not rediscover them:

• The BFF must be DOWN when a chunk starts. eval-env.config.ts clones oshun_dev → oshun_eval at global setup, and Postgres refuses CREATE DATABASE … TEMPLATE while any connection holds the source — a running BFF is exactly such a holder. The chunk then runs with "workbench families stay blocked" and silently grades a narrower deck. Boot the BFF AFTER the clone line appears in the log, or between chunks.
• The runnable deck file is assistant-golden.eval.ts. The deck-case sources (deck-*-cases.ts) are data, not test files; battery-transcripts.eval.ts is the judge-corpus capture. The eval:assistant and eval:assistant:battery package scripts now select those files separately, so a deck run cannot spend on or overwrite the labeling corpus.
• Foreground chunks that overflow the 560s window survive in background inconsistently. Grade lines print per case as they finish, so a killed run's completed cases salvage from the log (grep 'ADVISORY') — append to the pool and relaunch only the missing ids, never regrade salvaged ones.

Completion re-audit (2026-08-28)#

All seven docs/workbench family cases still exist, and every one still has a dependency/verdict row above. Later phases resolved the gaps exactly along the lines this note supplied as design input:

• the eval global setup now points operators at a checked-in frozen full-corpus slice while preserving the metadata-only member corpus (pages: 0, chunks: 0) by design;
• workbench cases receive a disposable eval clone when the operator has not supplied a database URL; the clone is run-unique, deterministically seeds the minimum ledger-backed state, and is force-dropped by global teardown; setup failures are narrated rather than silently graded as a narrower surface;
• family-docs-member-scope now grades honest tool-less absence; family-wbr-explorer-link has the superset-allowlist repair and remains advisory at its measured 9/10; family-wbr-member-refusal remains correctly environment-independent.

eval-env-notes.spec.ts now locks the note to the live family-case inventory and checks both corpus fixtures (full nonempty, member semantically empty). This prevents a new docs/workbench family case from inheriting an unearned "blocked" label or acquiring an unnamed fixture dependency.

eval-runner-isolation.spec.ts also pins the separate deck and transcript entry points. A partial deck selector can no longer fail an unrelated capture suite, and a routine deck run can no longer rewrite human-labeling artifacts.