Eve library

RUN-002 evidence pack — durable append, checkpoint, migration, and replay

Evidence

Revision 1

On this page

Work-package charter#

  • Task ID: RUN-002
  • Owner: Oshun common run-protocol durability maintainer; implementation owner: Codex
  • Accountable product: Shared V1–V9 autonomous-content platform
  • Priority: P0
  • Target milestone: 2026-07-15 SOTA gap-closure program
  • Risk tier: Critical persistence-ordering, recovery, concurrency, idempotency, audit-history, and release-integrity risk
  • Dependencies: RUN-001 strict common aggregate and deterministic attempt identity; existing common durable-job and V8 stage-journal concepts; AUD-003 reproducible verification; AUD-006 generated capability truth
  • Affected schemas and runtime: Hash-chained durable run events, V1/V2 checkpoints, forward/backward migration, retention-bearing archive segments, storage interface, file-backed crash-recoverable reference adapter, cross-process conformance and SIGKILL matrix, root/namespace/subpath exports, and focused CI
  • Rollout strategy: Use the portable protocol as the conformance target and the file adapter as the process-boundary reference implementation. Add product database/worker adapters under RUN-004; do not replace a product store merely because this reference adapter exists.
  • Rollback strategy: Revert the durable protocol, file adapter, tests, exports, CI expansion, audit qualification, and ledger closure together. Existing product orchestration remains unchanged and therefore needs no data rollback for RUN-002.
  • Evidence-pack location: This file

Durable event and write-before-effect protocol#

Every durable event carries a schema version, stable run/thread identity, store-assigned sequence, previous-event digest, event kind, timestamp, transition reference, idempotency key, optional effect boundary, state or attempt-allocation payload, explicit retention references, canonical payload digest, derived event ID, and event digest. Strict parsing recomputes state, payload, identity, and event hashes. Replay rejects sequence gaps, broken hash links, duplicate event/idempotency identities, time regression, changed immutable run identity, nondeterministic attempt identity, and noncontiguous attempt ordinals.

persistBeforeVisibleEffect awaits a durably appended effect-intent event before invoking a tool/publication callback. A failed optimistic append never invokes the callback. This proves the intent-before-visible-effect ordering barrier; RUN-003 still owns provider execution receipts, unknown-outcome reconciliation, and end-to-end effect deduplication.

Atomic file-backed reference adapter#

FileDurableRunStore persists a strict store snapshot and uses an atomic lock directory as its inter-process transaction boundary. Each mutation acquires the lock, reloads and validates the latest complete hash chain, checks the expected head, allocates sequence or attempt ordinal inside the transaction, writes a mode-0600 temporary file, fsyncs it, atomically renames it, fsyncs the parent directory, and releases the lock. Stale lock recovery is bounded and lock acquisition has a timeout.

Two independently constructed adapter instances prove same-key deduplication, one-winner optimistic concurrency for distinct writers, monotonic sequence allocation, and contiguous deterministic attempt ordinals. Replica append accepts only a continuation of the complete history and rejects duplicate or nonmonotonic input. The file adapter is an actual process-boundary reference store, not a claim of multi-host database, distributed-consensus, or product runtime integration.

Checkpoints, migrations, stable resume, and replay#

The portable protocol defines strict checkpoint schema versions 1 and 2. Forward migration maps legacy thread/cursor/head fields into the current stable thread ID, sequence cursor, state digest, parent checkpoint, and retention manifest form. Backward migration emits the strict V1 representation. Tests exercise V2 to V1 to V2 compatibility, preserved state/cursor/head identity, unknown-field rejection, and unsupported-version rejection.

Checkpoint creation binds materialized RUN-001 state to the exact state-bearing event cursor and digest. Resume looks up the run by stable thread ID, loads the latest checkpoint, replays later events, independently replays from origin, and fails if the results disagree. A real second Node process opens the file and recovers the checkpointed run through the same stable thread ID.

Process-death recovery matrix#

The adapter exposes six explicit conformance fault boundaries:

  1. after lock acquisition;
  2. after latest-state load;
  3. before temporary-file write;
  4. after temporary-file fsync;
  5. after atomic rename; and
  6. after parent-directory fsync.

For each boundary, the test launches a separate Node process, attempts one queued-to-planning transition, sends the process SIGKILL at that exact point, and opens the same store from a recovery adapter. Every case yields either the prior queued state or the committed planning state, never a partial record and never fabricated success. The recovered store then accepts the correct next transition. This is exhaustive for the adapter's durable commit boundaries; it is not a power-loss, kernel/filesystem fault, network partition, or product pipeline crash certification.

Compaction, archive, and retention#

Compaction first creates or reuses an integrity-bound checkpoint, then moves the eligible active prefix into a strict archive segment. The segment retains the original events and hash chain, cursor/head digests, archive digest, and a manifest for parent/fork lineage, approval records, legal records, and learning attribution. Archive import validates the internal chain and refuses a retention manifest that omits event-pinned references.

readHistory merges archived and active events in the original order, and resume after compaction must equal origin replay. The conformance test archives all active events while proving that every event digest and every lineage, approval, legal, and learning reference remains available. The reference file keeps archive segments inline for auditability; it does not claim external cold-storage lifecycle, retention deletion, encryption, or reduced total file size.

Artifact identity#

Artifact SHA-256
libs/contracts/src/run-protocol/run-model.ts 1d210cd89e78409c0a2c73804fba391e118388f6fddc183c38e6c1b6cf110f4b
libs/contracts/src/run-protocol/durable-run-protocol.ts 99ccbd7627b651eaa6e08516c60541c06798edc1fba527dbca212159a28f6568
libs/contracts/src/run-protocol/file-durable-run-store.ts cd4799321d826302459ddd65a02c6b408a93342e14df84b42ed4dd6530bb76a1
libs/contracts/src/run-protocol/durable-run-protocol.spec.ts 0ae481f213895bb5bc2e4168d5c86d47098efb6f6dbd87a938f2c62de6811878
libs/contracts/src/run-protocol/index.ts cf6855e981df59dda68ae2939bf3d16ae750a299cffe7284c891a50340066f97
package.json 60ccf0a12c4cc16cd76e1d4b381ee86df07198bd2443e0e003ed389f449e266e
.github/workflows/run-protocol-contract.yml 930c1394f7b84d9610f2681c54f8a58a26b91cfb6ab841a61204ff3339c0b3a8

Automated and adversarial verification#

  • pnpm verify:contracts:run-protocol passes targeted ESLint with zero warnings, 33/33 RUN-001/RUN-002 Vitest cases with one worker, and contracts source typechecking. RUN-002 contributes 15 focused cases.
  • The RUN-002 suite proves canonical thread/event identities, strict unknown field and digest rejection, duplicate/nonmonotonic/broken-chain rejection, archive integrity, fresh-adapter recovery, write-before-effect ordering, identical-request deduplication, idempotency conflict rejection, cross-instance atomic sequence/attempt allocation, optimistic race handling, strict replica continuation, forward/backward checkpoint migration, an actual second-process checkpoint resume, all six SIGKILL commit boundaries, lossless archive/retention replay, on-disk corruption rejection, and stable repeated digests.
  • Existing common durability compatibility remains green at 54/54 durable-job-orchestration and job-durability-recovery tests.
  • pnpm verify:contracts:content-slot remains green at 17/17, with lint and typechecking, because durable state continues to validate the RUN-001 content-slot/risk/autonomy aggregate.
  • NX_DAEMON=false pnpm exec nx run @oshun/contracts:build --parallel=1 passes both before and after the crash-matrix extension. Runtime probes verify the checkpoint/event versions, file adapter, and effect barrier through the root flat export, RunProtocolContracts namespace, and @oshun/contracts/run-protocol subpath.
  • Resource gates observed about 10–11 GiB available RAM, 30 GiB free swap, and 38 GiB free disk. Each package build ran alone at parallelism one and was polled at least every 30 seconds.
  • Source pre-commit ESLint, Prettier, stub-indicator, asset locking, and V5 documentation checks passed. The prohibited broad workspace typecheck was bypassed with SKIP_TYPECHECK=1 only after focused contracts typechecking, tests, and resource-gated builds passed.

Negative proof and task boundary#

  • No V1–V9 product pipeline writes this store, no product database implements the interface, no schema migration creates a production table, and no deployed worker resumes a common run. RUN-004 owns adapters/conformance and RUN-005 owns scheduler/runtime integration.
  • The reference adapter serializes writers with one shared-filesystem lock. It does not prove multi-host consensus, remote object-store atomicity, database isolation, high availability, regional failover, backup restore, encryption, key rotation, or production scale/performance.
  • SIGKILL proves process-death behavior on this Linux host, not sudden power loss or adversarial filesystem/kernel corruption. Fsync and atomic rename are implemented, but storage-platform durability still requires its own fault certification.
  • Effect intent is durable before the callback, but provider invocation, authoritative result persistence, timeout/unknown reconciliation, target ownership rules, and exactly-once user-visible behavior remain RUN-003.
  • Archive compaction reduces the active replay set but keeps archived events in the same reference file. External archive placement and retention/deletion policy remain unimplemented.
  • The fixture is synthetic contract/reference-adapter proof, not a production run, product publication, tool mutation, legal record, learning event, or operational recovery incident.
  • No user-facing UI, mobile flow, media/model pipeline, database migration, or Unreal asset changed. Playwright/mobile/browser accessibility, provider/media/database, and Unreal execution are therefore inapplicable to RUN-002.

Publication identity#

  • Initial durable-spine source commit: b957521d6a89de13d82bf8b5095682cdc8b422e8
  • Crash-recovery completion source commit: 754a4ec55c812ecffc9c244169301a435501083f
  • First remote-integrated tested complete source: 754a4ec55c812ecffc9c244169301a435501083f
  • Source-documentation alignment commit: 10011dcd67900d8c1c8f422d68cd2e358500f549
  • Publication metadata commit: a38ba5ca50954e474d2363ac396693d6aa541a99