# Eve internal engineering benchmark v1.0.0 — Task 11.1

- **Initiative:** `eve-sota-gap-closure-2026-09-01`
- **Task:** 11.1
- **Frozen:** 2026-09-12
- **Machine contract:**
  [`eve-engineering-benchmark.schema.json`](eve-engineering-benchmark.schema.json)
- **Release:**
  [`eve-engineering-benchmark/v1.0.0/release.json`](eve-engineering-benchmark/v1.0.0/release.json)
- **Verifier:**
  [`../../tools/eve-everywhere/verify-engineering-benchmark.mjs`](../../tools/eve-everywhere/verify-engineering-benchmark.mjs)

Task 11.1 now has a frozen, content-addressed benchmark made from eleven real
pieces of work accepted on `origin/main`. The cases cover all ten required
software-delivery classes; the combined DCC/native class has two independent
cases so a Blender result cannot stand in for native desktop control, or vice
versa. Every oracle is a one-parent commit, every candidate snapshot is that
commit's exact parent, and Git blob, tree, and raw change-set digests bind the
grader to the historical work rather than to a prose reconstruction.

This record establishes the corpus and its context boundary. It does **not** run
an autonomous engineering cohort or claim success, cost, time, intervention,
rollback, or quality results. Tasks 11.3–11.6 own those later execution and
measurement boundaries.

## Version contents

| Case         | Class               | Variant        | Size   | Historical work shape                                     |
| ------------ | ------------------- | -------------- | ------ | --------------------------------------------------------- |
| `eng-v1-001` | diagnosis-only      | repository     | medium | capability-fabrication audit with one weakness left open  |
| `eng-v1-002` | bug-fix             | repository     | small  | MP4 version-1 track-header dimension repair               |
| `eng-v1-003` | feature             | repository     | medium | authoritative release-aware evaluation selection          |
| `eng-v1-004` | refactor            | repository     | large  | obsolete package retirement with behavior-preserving move |
| `eng-v1-005` | migration-contract  | service        | large  | five-phase persisted aggregate migration                  |
| `eng-v1-006` | security-repair     | service        | large  | memory injection, admission, and deletion fencing         |
| `eng-v1-007` | docs-product-graph  | repository     | large  | product-graph closeout and fail-loud corpus walk          |
| `eng-v1-008` | UI/Playwright       | browser        | medium | live-region repair and keyboard-only browser journey      |
| `eng-v1-009` | service-integration | service        | large  | real local-platform LTI launch handshake                  |
| `eng-v1-010` | DCC/native          | DCC            | medium | real Blender stdin-RPC save and export smoke              |
| `eng-v1-011` | DCC/native          | native desktop | large  | native input, capture, accessibility, and privacy fixture |

The candidate-readable catalog is
[`public/catalog.json`](eve-engineering-benchmark/v1.0.0/public/catalog.json).
It contains the task brief, constraints, task class, runtime needs, and exact
base snapshot. It contains no oracle commit, source provenance, criterion,
verification command, or evaluator canary. All v1 cases are held out; none may
be used for prompting, tuning, routing changes, checker changes, or agent
instructions and then reused as an unseen result.

The evaluator acceptance record is deliberately separated at
`eve-engineering-benchmark/v1.0.0/evaluator/acceptance.json`. It contains the
accepted commit, pinned historical source blobs, exact criteria, verification
commands, and unique leak canaries. It is internal QA material, not a candidate
resource. Result logs may retain criterion IDs and verdicts, never criterion
text, oracle locations, or canaries.

## Candidate context and contamination boundary

The boundary is mechanical, not a naming convention:

1. Select one case and materialize only a `git archive` of its historical base
   commit. An archive contains no `.git` directory, refs, reflogs, remote URL,
   oracle commit, current benchmark files, or forward history.
2. Give the candidate only the JSON emitted by
   `export-engineering-benchmark-context.mjs --case=<id>`. That program has one
   file read and it targets the public catalog. The semantic verifier checks the
   emitted keys and scans every case output for every oracle commit, tree,
   change-set digest, commit title, and leak canary.
3. Deny network access and do not mount the current checkout, Git object store,
   other case prompts, evaluator directory, or prior run outputs. A policy
   breach invalidates the run; it is not repaired in place.
4. End candidate access before the QA lane mounts the evaluator record and
   grades the exact resulting artifact. The implementer cannot author or alter
   the evidence used for its verdict.

The verifier also rejects a base that already contains its oracle, a base tree
that contains this benchmark, an oracle not reachable from `origin/main`, a
merge commit, duplicated work, stale blob/tree/change-set hashes, a copied
commit title, weakened network/history/case-exposure policy, or mismatched
public/evaluator case identities.

Because the hidden record is committed for internal QA reproducibility, running
a candidate in the current repository checkout is contamination by definition.
The supported candidate context is the historical archive above. A host with
root access could deliberately bypass mounts and network policy; Task 11.3 must
enforce those execution-lane controls and retain the runtime receipt before a
Task 11.6 result can be admitted.

## Security, privacy, and data rights

The corpus stores no raw patch, raw conversation, member record, credential
value, provider response, database URL, launch token, cookie, screenshot, or
generated media. Public briefs are paraphrases. Evaluator provenance retains
commit and path identities plus content digests, not commit bodies or author
identities. Each case declares repository-owned source rights, no personal data,
no credential values, no raw conversations, its redaction decision, and the
versioned internal retention purpose.

The verifier scans both records for private-key markers, JWT shapes, common
credential prefixes, credential-bearing URLs, and email addresses. These are
corpus hygiene checks, not a general secret scanner. Historical base workspaces
still contain the repository as it existed at the selected commit and therefore
must receive the same secret-safe execution controls as ordinary engineering
work.

## Versioning and rotation

`release.json` pins the byte length and SHA-256 of both the public and evaluator
documents and composes them into one benchmark digest.
`seal-engineering-benchmark.mjs --check` reconstructs every Git-derived field
and fails on drift. Once results are visible, v1.0.0 is immutable: any
correction or case change is a new version, with its rationale fixed before
results are inspected.

If a case, criterion, oracle, or grader detail reaches a candidate—or any
prompt, skill, router, checker, or implementation is changed with knowledge of
its answer—the affected case is invalidated and retired. It may enter a visible
training/diagnostic pool, but a fresh closed-work case must replace it before
the held-out class can be measured again. External model pretraining and caches
cannot be disproved by this repository; known cases are rejected at admission,
and suspected knowledge requires retirement, not a favorable interpretation.

Task 12.5 still owns the broader source/licence, split, freshness, retention,
and review lifecycle for all evaluation data. Task 12.2 still owns sample-size
and statistical feasibility. This benchmark does not pre-empt either task.

## Executed controls

The focused suite exercises the real CLI and rejects all of these mutations:

- evaluator canary or oracle reference in candidate material;
- a full-history workspace, enabled network, or all-case context;
- an omitted required class or either half of DCC/native;
- an oracle already present in the base snapshot;
- a stale source blob or release digest;
- duplicated historical work;
- missing privacy review;
- a task brief copied from the closed commit title; and
- mounting evaluator acceptance into the candidate boundary.

The post-control verifier rerun is green. Draft 2020-12 validation, sealer
freshness, Node syntax, ESLint, Prettier, the gap/task/evidence matrix, generic
evidence-manifest admission, and docs-center rendering are part of the filed
Task 11.1 receipt.