Disciplines · Audits

Eve engineering plan-quality contract v1.0.0 — Task 11.2

The weights total 100.

7sections6 minread
Markdown source
On this page

Task 11.2 now has a frozen, deterministic rubric for engineering-plan quality. It grades the plan that exists before work begins; it does not infer quality from a later successful commit, an agent narrative, or a green command. A plan passes only when every material goal clause is covered or validly excluded, every dimension scores at least 3 of 4, the weighted score is at least 85, and there are no blockers.

This is a rubric, schema, and evaluator. It does not implement durable goal or plan revisions, persistence, authenticated planning APIs, concurrency, leases, review, shipping, or verification-result storage. Tasks 11.9–11.11 own those production capabilities under ADR-0076. A passing Task 11.2 grade cannot be cited as evidence that any of them exists.

Frozen dimensions and admission bar#

Dimension Weight Minimum What must be directly present
Requirements and scope 20 3 Stable requirements, testable criteria, total goal-clause coverage, accepted exclusions, non-goals, contained paths
Authoritative source discovery 15 3 A current authoritative source and authority basis for every requirement
Dependency rationale 15 3 Exact work nodes, typed acyclic edges, per-edge reasons, explicit independent-node reasons, no orphan refs
Risk analysis 15 3 High-risk coverage, effects, triggers, mitigations, reversibility, irreversible authority
Verification matrix 20 3 Every requirement and work item bound to direct proof at the required boundary, with high-risk red controls
Rollback readiness 10 3 Trigger, steps, verification, and authorized compensation for every work item
Clarification threshold 5 3 Material ambiguity and conflicts stop; only attributed low-impact assumptions continue

The weights total 100. Strong prose in six dimensions cannot average away a zero in the seventh: any blocker makes the result non-passing regardless of the arithmetic total. not-machine-checkable remains an honest open result and does not close a requirement.

Traceability and source authority#

The plan input binds exact goalRevisionId, goal digest, decision-set digest, repository commit, stable requirement IDs, goal-clause refs, accepted decision refs, source observations, work items, dependency edges, verification rows, and rollback rows. The grading context supplies the current versions. Any mismatch in goal revision, goal digest, decision digest, or repository snapshot yields PLAN-STALE; the grader never silently upgrades the old plan to the new goal.

Every requirement must cite a source observed as authoritative in the current context. The source row records its locator, content digest, authority class, and why it governs that requirement. A source being nearby, recent, or useful does not make it authoritative. Supporting sources may be retained, but they cannot satisfy the authoritative-source requirement. Requirement/source refs must agree in both directions, and duplicate or orphan identities are rejected. The Task 11.2 fixture anchors itself to the initiative ledger, ADR-0076, and the generated gap/task/evidence matrix.

Non-goals are similarly bidirectional: a non-goal must name a current candidate exclusion, that exclusion must cite an accepted human decision, and the same clause cannot also be planned as required work. Scope prefixes must be normalized repository-relative POSIX paths, preventing parent traversal or lookalike directory prefixes from passing containment checks.

The dependency graph is evaluated separately from requirement coverage. Its node set must equal the work-item set. Every edge has a stable ID, type, and individual rationale; each work item lists the exact incoming edge IDs and also explains why its dependency position is correct. Unknown nodes, orphan refs, duplicate edges, and cycles are rejected.

Named failure classes#

The conformance suite preregisters one complete plan and seven independent faults. Each mutation is applied to a deep copy, graded through the same public function as the CLI, and checked for its expected outcome and issue code.

Fault Required code Outcome
Required goal clause left uncovered REQ-UNCOVERED revise
Two current accepted decisions conflict DEC-CONTRADICTORY clarify
A work item enters forbidden/non-goal path SCOPE-WRONG revise
Proof is indirect or at the wrong boundary PROOF-WEAK revise
Goal, decision, or repository base is old PLAN-STALE stale
A work item has no complete rollback row ROLLBACK-MISSING revise
Material ambiguity is assumed through CLARIFICATION-REQUIRED clarify

The semantic verifier adds red controls for a removed rubric dimension, weakened pass thresholds, a weakened schema boundary, stale live source digest, a missed performance floor reported as passing, and a fabricated runtime-capability claim. Every control must fail through the real verifier CLI, followed by an unmodified green regression.

Clarification threshold#

The frozen threshold is impact-based:

  • A low-impact ambiguity may proceed only as an explicit assumed-low record with its rationale. It remains visible to review.
  • Material ambiguity must be clarified. A resolution that changes authority or scope needs an accepted human decision; model confidence and recency are not authority.
  • Conflicting current human decisions, missing authority, material goal or non-goal changes, requirement removal, dependency bypass, proof waiver, irreversible action, cloud spend/live autonomy, and an uncertain destructive target always stop for clarification.

These rules deliberately distinguish a useful question from reflexively asking the operator about every harmless detail. The grader rejects both silent material assumptions and a plan that weakens the declared threshold.

Risk, proof, and rollback#

High and critical requirements require risk rows and verification negative controls. A risk names effect, trigger, mitigation, reversibility, and—when irreversible—the accepted human authorization. The verification matrix binds requirements and work items to an exact proof boundary, executable command or human rubric, expected outcome, directness, negative control, and limitations. Its reverse refs must resolve, and a high-risk negative control must be attached to a direct row at the required boundary rather than to unrelated indirect evidence.

Every work item names one rollback row. Reversible work needs a trigger, steps, and verification. Irreversible work additionally needs a compensation and an accepted human authorization. “Revert if needed,” status rewriting, or a green process exit is not rollback evidence.

Measured evaluator quality#

The preregistered measurement requires all eight conformance cases detected, 10,000 in-process grades, p95 latency below 25 milliseconds per 50-grade batch, and RSS growth no greater than 64 MiB. The retained measurement-2026-09-12.json records the Node version, observed latency distribution, memory bounds, and per-case outcomes. These are evaluator conformance and resource measurements, not engineering-agent success, task latency, or delivery cost.

Limitations and downstream ownership#

  • The fixture's human-decision shape is input data, not authenticated identity proof. Task 11.9 owns versioned goals, requirements, decisions, persistence, APIs, tenant isolation, and concurrency.
  • The dependency and verification rows are graded documents, not durable graph entities. Task 11.10 owns their persistence, canonical digests, APIs, and competing revisions.
  • No work item is made ready, leased, reviewed, shipped, or verified here. Task 11.11 owns the real lifecycle binding and stale-plan enforcement at those transitions.
  • The deterministic conformance fixture is visible and cannot stand in for the hidden real-work benchmark. Tasks 11.4–11.6 own independent review and unseen outcome measurement.
  • Phase 11 and G12 remain open.