Eve internal engineering benchmark v1.0.0 — Task 11.1

• Initiative: eve-sota-gap-closure-2026-09-01
• Task: 11.1
• Frozen: 2026-09-12
• Machine contract: eve-engineering-benchmark.schema.json
• Release: eve-engineering-benchmark/v1.0.0/release.json
• Verifier: ../../tools/eve-everywhere/verify-engineering-benchmark.mjs

Task 11.1 now has a frozen, content-addressed benchmark made from eleven real pieces of work accepted on origin/main. The cases cover all ten required software-delivery classes; the combined DCC/native class has two independent cases so a Blender result cannot stand in for native desktop control, or vice versa. Every oracle is a one-parent commit, every candidate snapshot is that commit's exact parent, and Git blob, tree, and raw change-set digests bind the grader to the historical work rather than to a prose reconstruction.

This record establishes the corpus and its context boundary. It does not run an autonomous engineering cohort or claim success, cost, time, intervention, rollback, or quality results. Tasks 11.3–11.6 own those later execution and measurement boundaries.

Version contents#

The candidate-readable catalog is public/catalog.json. It contains the task brief, constraints, task class, runtime needs, and exact base snapshot. It contains no oracle commit, source provenance, criterion, verification command, or evaluator canary. All v1 cases are held out; none may be used for prompting, tuning, routing changes, checker changes, or agent instructions and then reused as an unseen result.

The evaluator acceptance record is deliberately separated at eve-engineering-benchmark/v1.0.0/evaluator/acceptance.json. It contains the accepted commit, pinned historical source blobs, exact criteria, verification commands, and unique leak canaries. It is internal QA material, not a candidate resource. Result logs may retain criterion IDs and verdicts, never criterion text, oracle locations, or canaries.

Candidate context and contamination boundary#

The boundary is mechanical, not a naming convention:

1. Select one case and materialize only a git archive of its historical base commit. An archive contains no .git directory, refs, reflogs, remote URL, oracle commit, current benchmark files, or forward history.
2. Give the candidate only the JSON emitted by export-engineering-benchmark-context.mjs --case=<id>. That program has one file read and it targets the public catalog. The semantic verifier checks the emitted keys and scans every case output for every oracle commit, tree, change-set digest, commit title, and leak canary.
3. Deny network access and do not mount the current checkout, Git object store, other case prompts, evaluator directory, or prior run outputs. A policy breach invalidates the run; it is not repaired in place.
4. End candidate access before the QA lane mounts the evaluator record and grades the exact resulting artifact. The implementer cannot author or alter the evidence used for its verdict.

The verifier also rejects a base that already contains its oracle, a base tree that contains this benchmark, an oracle not reachable from origin/main, a merge commit, duplicated work, stale blob/tree/change-set hashes, a copied commit title, weakened network/history/case-exposure policy, or mismatched public/evaluator case identities.

Because the hidden record is committed for internal QA reproducibility, running a candidate in the current repository checkout is contamination by definition. The supported candidate context is the historical archive above. A host with root access could deliberately bypass mounts and network policy; Task 11.3 must enforce those execution-lane controls and retain the runtime receipt before a Task 11.6 result can be admitted.

Security, privacy, and data rights#

The corpus stores no raw patch, raw conversation, member record, credential value, provider response, database URL, launch token, cookie, screenshot, or generated media. Public briefs are paraphrases. Evaluator provenance retains commit and path identities plus content digests, not commit bodies or author identities. Each case declares repository-owned source rights, no personal data, no credential values, no raw conversations, its redaction decision, and the versioned internal retention purpose.

The verifier scans both records for private-key markers, JWT shapes, common credential prefixes, credential-bearing URLs, and email addresses. These are corpus hygiene checks, not a general secret scanner. Historical base workspaces still contain the repository as it existed at the selected commit and therefore must receive the same secret-safe execution controls as ordinary engineering work.

Versioning and rotation#

release.json pins the byte length and SHA-256 of both the public and evaluator documents and composes them into one benchmark digest. seal-engineering-benchmark.mjs --check reconstructs every Git-derived field and fails on drift. Once results are visible, v1.0.0 is immutable: any correction or case change is a new version, with its rationale fixed before results are inspected.

If a case, criterion, oracle, or grader detail reaches a candidate—or any prompt, skill, router, checker, or implementation is changed with knowledge of its answer—the affected case is invalidated and retired. It may enter a visible training/diagnostic pool, but a fresh closed-work case must replace it before the held-out class can be measured again. External model pretraining and caches cannot be disproved by this repository; known cases are rejected at admission, and suspected knowledge requires retirement, not a favorable interpretation.

Task 12.5 still owns the broader source/licence, split, freshness, retention, and review lifecycle for all evaluation data. Task 12.2 still owns sample-size and statistical feasibility. This benchmark does not pre-empt either task.

Executed controls#

The focused suite exercises the real CLI and rejects all of these mutations:

• evaluator canary or oracle reference in candidate material;
• a full-history workspace, enabled network, or all-case context;
• an omitted required class or either half of DCC/native;
• an oracle already present in the base snapshot;
• a stale source blob or release digest;
• duplicated historical work;
• missing privacy review;
• a task brief copied from the closed commit title; and
• mounting evaluator acceptance into the candidate boundary.

The post-control verifier rerun is green. Draft 2020-12 validation, sealer freshness, Node syntax, ESLint, Prettier, the gap/task/evidence matrix, generic evidence-manifest admission, and docs-center rendering are part of the filed Task 11.1 receipt.