- Task: 2.3
- Evaluated: 2026-09-04
- Decision: bounded-resumable-drain-implemented
- ADR gate:
bounded-resumable-drain(docs/adr/ADR-0076-eve-governed-delivery-lifecycle.md) - Run record version:
eve.fleet-drain-run.v1 - Record digest:
7cfa3814d1826453f84f993b60c687fcdb1063b8654b846da2fbe29da39a37fe
The rule#
The drain is selection and supervision, never a second lifecycle. It reads the queue, gates each item on a fresh host reading, spawns the agent in its own process group under a hard timeout, reaps the group, and then reads the work item back to learn what happened. It never leases, reports, or ships — those remain the agent’s own attributed MCP calls — and it never infers an outcome from a process exit code.
Every capability in this record binds to a symbol the library exports, a marker the CLI contains, and a proof that exists by name; the parity property is proved by the absence of any lease, report, or ship reference in the drain source and by the request log of a real HTTP workbench.
The eight capabilities, and what proves each#
| # | Capability | Requirement | Proved by |
|---|---|---|---|
| 1 | dry-run |
A dry run reports the plan, the budget, the next item, and the resource verdict without spawning an agent or writing anything to the intent plane. | test: the real CLI refuses to start when preflight cannot see a token |
| 2 | item-limit |
--limit is required and bounded at both ends; a drain with no explicit item budget is refused rather than defaulted. | test: --limit is required, because a drain without one is the unbounded loop; test: every numeric option is bounded at both ends; test: the real CLI refuses a missing --limit with a non-zero exit |
| 3 | resumable-run-id |
A run is identified, checkpointed to disk after every state change, and resumable; a resumed run keeps its ORIGINAL item budget and retains the attempts already made. | test: a resumed run keeps its original budget and cannot buy a second one; test: a run record from another schema version is refused, not coerced; probe: crash-and-resume |
| 4 | process-group-tracking |
The agent runs in its own process group; the pid and pgid are checkpointed BEFORE the spawn returns, and a resume reaps a group a dead run left behind. | test: the agent runs in its OWN process group, and the group outlives nothing; test: a resume surfaces the process group a dead run left behind; probe: crash-and-resume |
| 5 | clean-termination |
A per-item timeout and an operator signal both end the whole agent process group, escalating SIGTERM to SIGKILL, and everything the agent spawned dies with it. | test: a hung agent is killed at its timeout, and so is the grandchild it left running; test: killGroup tolerates a group that already ended and refuses pid 1 |
| 6 | per-item-resource-checkpoint |
Every item is gated on a fresh host reading before the spawn; the gate reports every closed reason, refuses an unreadable value rather than assuming headroom, and a supplied reading is marked as injected in the record. | test: the gate reports EVERY closed reason, not the first; test: an unreadable reading refuses instead of passing; test: the memory floors are the ones AGENTS.md states, and say where they came from; probe: resource-gate |
| 7 | version-and-capability-preflight |
Before any item, the drain confirms the token, the queue, the agent CLI version, the queue-semantics contract version, a writable state directory, and a clean supervised repository; a missing observation refuses rather than passes. | test: preflight admits a ready host and refuses each missing capability by name; test: preflight never treats a missing observation as a pass |
| 8 | no-unbounded-loop |
The loop is bounded four independent ways — item budget, wall-clock budget, queue exhaustion, and at most one attempt per item per run — and an unreadable clock exhausts the budget rather than running forever. | test: the loop is bounded four independent ways, each named; test: an unreadable clock exhausts the budget rather than running forever; probe: bounded-drain |
The parity boundary#
The drain may call only these, and the end-to-end probe 500s on anything else so a forbidden call would be loud:
GET /v1/workbench/queueGET /v1/workbench/items/:id/briefPOST /v1/workbench/verify
It may never call these, and its own source contains 0 reference(s) to them:
POST /v1/workbench/items/:id/leasePOST /v1/workbench/items/:id/reportPOST /v1/workbench/items/:id/shipped
Why the loop cannot run away#
- item budget (--limit, required)
- wall-clock budget (--budget-seconds)
- queue exhaustion
- at most one attempt per item per run
- Per item: bounded by --item-timeout-seconds, never unset
Resource gate#
| Threshold | Bytes | Source |
|---|---|---|
| Available memory floor | 2147483648 | AGENTS.md |
| Free memory floor (no swap) | 536870912 | AGENTS.md |
| Free disk floor | 2147483648 | this contract |
End-to-end probe#
Scenarios: bounded-drain, exit-zero-no-progress, exit-nonzero-shipped,
crash-and-resume, resource-gate. Negative controls:
exit-code-means-shipped, resume-buys-new-budget, ignore-resource-gate.
Real HTTP workbench: true. Real process groups: true. Clean git fixture: true.
Honest limits#
- This record closes task 2.3 only. It does not add typed triage (2.4), the operator read model (2.5), execution isolation (2.6), the orchestrator model and parity proof (2.7), or any live drain or soak result (2.8).
- No real coding agent has been driven by this drain. Every run in evidence used a controlled agent script against a local HTTP workbench that answers the three permitted operations; a real BFF, a real Codex session, and a real work item are task 2.8.
- The end-to-end probe supplies the per-item resource reading, because this
development host always carries a competing agent process that the gate
correctly refuses. Every such run is stamped
injected:in its record and keeps the real measurement beside it, so an injected run cannot be read as a measured one. - Execution isolation is preflight-level only: the drain observes that the supervised repository is clean and names it with --repo. Branch and worktree isolation, scoped credentials, command roots, and egress policy are task 2.6.
- The drain stops on the first item that does not advance, following the recorded recipe. It does not yet produce the distinct evidence-bearing triage record task 2.4 requires; it records the outcome and the reason and stops.
- The disk floor in the resource policy is chosen by this contract rather than derived from AGENTS.md, which requires the df check but names no threshold.