Eve library

RUN-003 evidence pack — idempotent and reconcilable side effects

Evidence

Revision 1

On this page

Work-package charter#

  • Task ID: RUN-003
  • Owner: Oshun common run-protocol effect-delivery maintainer; implementation owner: Codex
  • Accountable product: Shared V1–V9 autonomous-content platform
  • Priority: P0
  • Target milestone: 2026-07-15 SOTA gap-closure program
  • Risk tier: Critical mutation-deduplication, provider-ambiguity, publication-integrity, concurrency, ownership, and recovery risk
  • Dependencies: RUN-001 immutable aggregate and artifact lineage; RUN-002 durable append, atomic attempt allocation, replay, and shared-filesystem reference adapter; provider-specific idempotency and lookup guarantees
  • Affected schemas and runtime: Deterministic effect identity, provider delivery policies, hash-chained effect journal, effect reducer and storage interface, provider coordinator, file-backed reference adapter, root/ namespace/subpath exports, and focused CI
  • Rollout strategy: Treat the portable effect protocol and file adapter as the RUN-004 conformance target. A product adapter may declare only the delivery guarantees its provider actually offers, and an ambiguous effect fails closed unless provider reconciliation makes retry safe.
  • Rollback strategy: Revert the effect protocol, tests, file-store extension, exports, CI expansion, audit qualification, and ledger closure together. No product pipeline or database migration consumes this protocol yet, so RUN-003 has no product-data rollback.
  • Evidence-pack location: This file

Deterministic identity and ownership protection#

deriveDurableEffectIdentity binds the delivery key to tenant, immutable run identity, plan and step, durably allocated attempt ID and ordinal, tool, provider target and operation, and sorted artifact lineage with content and parent hashes. Tests vary the run, step, attempt, tool, target, and artifact content independently and prove that each changes the effect key.

Owner scope and mutation-class labels are deliberately excluded from the delivery key but included in the complete immutable effect-identity digest. This makes an owner-controlled relabel collide with the existing delivery identity instead of minting a second provider key. The store rejects that collision as an idempotency conflict. It also checks the effect against the registered run's tenant and immutable identity, a real plan/step, the exact store-allocated attempt tuple, the provider policy, and current artifact hashes and parent lineage before it accepts an intent.

The reference adapter cannot register a different immutable run under an existing stable run/thread identity. A changed run-identity digest is rejected before a new effect journal is created. Legitimate new attempts, targets, or artifact revisions receive new keys because they represent different mutations.

Durable journal and provider ordering#

Every effect record contains immutable identity and provider policy plus a strict, hash-chained journal. Events carry a stable event ID, effect key, contiguous revision, previous-event digest, monotonic timestamp, idempotency key, canonical detail digest, and event digest. Replay rejects a broken chain, duplicate event or idempotency identity, time regression, illegal transition, noncontiguous delivery attempt, mismatched retry authority, or changed effect identity.

The legal lifecycle is:

  1. persist immutable intent and request digest;
  2. durably claim a delivery attempt with worker and lease identity;
  3. call the provider with the exact effect key and any reconciled completed/ pending artifact set;
  4. persist the authoritative provider receipt and response digest, or persist an explicit unknown outcome;
  5. reconcile unknown, partial, or expired-dispatch state with provider truth;
  6. authorize retry only from the immediately preceding authoritative not-applied or resumable partial reconciliation; otherwise succeed, fail, wait, or block.

The provider callback observes the journal in dispatching state, proving both intent and claim were committed before invocation. Applied, partial, and rejected execution results require an authoritative receipt. Determinate lookup results require a receipt, partial results require pending artifacts, and no artifact may be both completed and pending.

Reconciliation and safe delivery modes#

Provider policy declares exactly-once-key, idempotent-retry, reconcile-before-retry, or at-most-once, plus lookup, partial-resume, and idempotency-field support. Keyed guarantees require an explicit provider key field; partial resume is valid only for a keyed idempotent policy.

Duplicate coordinator calls return terminal durable state without reinvoking the provider. A lost acknowledgment or timeout is first recorded as unknown. The next coordinator call must query the provider before it can proceed:

  • applied becomes durable success without redelivery;
  • not-applied creates retry authority and reuses the same effect key;
  • partial can resume only under a keyed policy that declares partial-resume support, with the reconciled completed and pending artifact sets passed to the adapter;
  • in-progress remains non-deliverable and will be reconciled again;
  • unknown, failed lookup, absent lookup support, or unsafe partial semantics blocks delivery instead of guessing.

An active durable worker lease prevents a concurrent worker from invoking the provider. After lease expiry, a failover worker reconciles provider truth before claiming the next contiguous delivery attempt. Optimistic revision checks and the shared-filesystem lock give one durable claim winner across independently constructed adapter instances.

Exactly-once behavior therefore depends on an authoritative provider honoring the supplied key. The conformance mock proves that the coordinator supplies one stable key and does not repeat a completed delivery; it is not evidence that a real publication provider offers exactly-once semantics. When authoritative provider truth cannot establish a safe path, the coordinator fails closed.

Reference-adapter persistence boundary#

FileDurableRunStore now implements both the RUN-002 run store and the RUN-003 effect store. Effect intents and journal events use the same locked, validated, temporary-write, file-fsync, atomic-rename, and directory-fsync transaction machinery as run events. Loading the store validates every effect identity and journal chain before returning data.

The effect journal is a separate record within the same file state, not an additional RUN-002 aggregate state event and not a database transaction spanning a real remote provider. The write-before-call barrier closes the local crash window before provider invocation; provider lookup and idempotency keys address the unavoidable ambiguity after invocation and before result persistence.

Automated scenario matrix#

Scenario Conformance result
Required key tuple Run, step, attempt ID/ordinal, tool, target, and artifact-lineage variants produce distinct keys.
Owner/run reuse Owner or mutation relabel collides and is rejected; changed immutable run identity is rejected.
Intent/result ordering Provider observes durable dispatch; authoritative result follows in the journal.
Duplicate delivery Repeated coordinator calls invoke the exactly-once mock once with one key.
Lost acknowledgment + restart A new adapter instance reconciles applied and never redelivers.
Timeout Retry occurs only after authoritative not-applied, using the original key.
Partial success Reconciled completed/pending artifacts reach the keyed resume attempt.
Unsafe ambiguity No lookup, failed lookup, or unsupported partial resume produces a durable block.
Worker crash/failover Expired claim is reconciled before failover delivery attempt two.
Concurrent workers Two adapter instances produce exactly one provider invocation and one durable success.
Retry-authority integrity A reason not matching the immediately preceding reconciliation is rejected.
Restart/idempotency integrity Journal digests survive adapter restart; conflicting event-key reuse is rejected.

Artifact identity#

Artifact SHA-256
libs/contracts/src/run-protocol/durable-effect-protocol.ts 5f5e90061109a07140123e77efaa8dbc05642172f8bae8d03b95f9e17447271b
libs/contracts/src/run-protocol/durable-effect-protocol.spec.ts dfd180f873b1e415e97ffa6b08e058c9627d319e8ab2d3b1d9bf5c788ce38af2
libs/contracts/src/run-protocol/file-durable-run-store.ts 670cab4b3df8c39cb2a4f35f0b815b7a4042f3f5b95333a6b641fc4f0b61f0c4
libs/contracts/src/run-protocol/index.ts cca361da05470e63afc1529f3180f93fa113689175179da3a491dd84da4f7673
libs/contracts/src/index.ts ee9f0f22297c7a4b29243221bf15e31290d6df75e479bcfe1bd7b83e39150f1d
package.json 3cf507687fef0e18b9168c23f785e19dbfa453f88447eacd7a8762e8d7652917
.github/workflows/run-protocol-contract.yml 930c1394f7b84d9610f2681c54f8a58a26b91cfb6ab841a61204ff3339c0b3a8

Automated and adversarial verification#

  • pnpm verify:contracts:run-protocol passes targeted ESLint with zero warnings, 45/45 RUN-001/RUN-002/RUN-003 Vitest cases with one worker, and contracts source typechecking. RUN-003 contributes 12 focused cases.
  • The RUN-003 suite covers all requested duplicate-delivery, lost- acknowledgment, timeout, partial-success, restart, failover, and concurrent-worker cases, plus key derivation, owner relabeling, durable ordering, unsafe-provider blocking, retry-authority integrity, and event idempotency conflicts.
  • Existing common durability compatibility remains green at 54/54 durable-job-orchestration and job-durability-recovery cases.
  • pnpm verify:contracts:content-slot remains green at 17/17, including lint and contracts typechecking, because effect ownership validates the RUN-001 content and artifact aggregate.
  • NX_DAEMON=false pnpm exec nx run @oshun/contracts:build --parallel=1 passes. Runtime probes verify the effect schema version, coordinator, identity derivation, and provider-result schema through the root flat export, RunProtocolContracts namespace, and @oshun/contracts/run-protocol subpath.
  • The build ran alone after the required resource gate. Available RAM remained 10–11 GiB, swap had about 30 GiB free, and disk had 38 GiB free.
  • Source pre-commit ESLint, Prettier, stub-indicator, asset locking, and V5 documentation checks passed. The prohibited broad workspace typecheck was bypassed with SKIP_TYPECHECK=1 only after focused contracts typechecking, tests, and the resource-gated build passed.

Negative proof and task boundary#

  • No V1–V9 product pipeline calls the coordinator, no product database implements the store, no migration persists effect records, and no deployed worker or publication provider has been exercised. RUN-004 owns product and database adapters; RUN-005 owns scheduler, approval, and runtime lifecycle integration.
  • The exactly-once provider is a deterministic conformance mock. There is no live provider receipt, placement observation, publication, rollback, canary, regional failover, or duplicate-delivery drill against production.
  • The file adapter is a shared-filesystem reference implementation. It does not prove multi-host consensus, database isolation, network partitions, regional high availability, backup restore, encryption, key rotation, or production throughput.
  • The provider call and local result append cannot be one atomic transaction. RUN-003 makes that boundary explicit and reconciles ambiguity, but correctness still relies on a provider truth surface and its declared key semantics.
  • The fixture is synthetic contract/reference-adapter proof, not a product run, real artifact release, tool mutation, legal record, learning event, or operational recovery incident.
  • No user-facing UI, mobile flow, media/model pipeline, database migration, or Unreal asset changed. Playwright/mobile/browser accessibility, provider/media/database, and Unreal execution are therefore inapplicable to RUN-003.

Publication identity#

  • Implementation and first remote-integrated tested source commit: aee2879220a22a0bc4402c5828a13acb135059a3
  • Publication metadata commit: 585ee7acd82e529ff54073ddced526e73e133f76