Work-package charter#
- Task ID:
RUN-002 - Owner: Oshun common run-protocol durability maintainer; implementation owner: Codex
- Accountable product: Shared V1–V9 autonomous-content platform
- Priority: P0
- Target milestone: 2026-07-15 SOTA gap-closure program
- Risk tier: Critical persistence-ordering, recovery, concurrency, idempotency, audit-history, and release-integrity risk
- Dependencies: RUN-001 strict common aggregate and deterministic attempt identity; existing common durable-job and V8 stage-journal concepts; AUD-003 reproducible verification; AUD-006 generated capability truth
- Affected schemas and runtime: Hash-chained durable run events, V1/V2
checkpoints, forward/backward migration, retention-bearing archive segments,
storage interface, file-backed crash-recoverable reference adapter,
cross-process conformance and
SIGKILLmatrix, root/namespace/subpath exports, and focused CI - Rollout strategy: Use the portable protocol as the conformance target and the file adapter as the process-boundary reference implementation. Add product database/worker adapters under RUN-004; do not replace a product store merely because this reference adapter exists.
- Rollback strategy: Revert the durable protocol, file adapter, tests, exports, CI expansion, audit qualification, and ledger closure together. Existing product orchestration remains unchanged and therefore needs no data rollback for RUN-002.
- Evidence-pack location: This file
Durable event and write-before-effect protocol#
Every durable event carries a schema version, stable run/thread identity, store-assigned sequence, previous-event digest, event kind, timestamp, transition reference, idempotency key, optional effect boundary, state or attempt-allocation payload, explicit retention references, canonical payload digest, derived event ID, and event digest. Strict parsing recomputes state, payload, identity, and event hashes. Replay rejects sequence gaps, broken hash links, duplicate event/idempotency identities, time regression, changed immutable run identity, nondeterministic attempt identity, and noncontiguous attempt ordinals.
persistBeforeVisibleEffect awaits a durably appended effect-intent event
before invoking a tool/publication callback. A failed optimistic append never
invokes the callback. This proves the intent-before-visible-effect ordering
barrier; RUN-003 still owns provider execution receipts, unknown-outcome
reconciliation, and end-to-end effect deduplication.
Atomic file-backed reference adapter#
FileDurableRunStore persists a strict store snapshot and uses an atomic lock
directory as its inter-process transaction boundary. Each mutation acquires the
lock, reloads and validates the latest complete hash chain, checks the expected
head, allocates sequence or attempt ordinal inside the transaction, writes a
mode-0600 temporary file, fsyncs it, atomically renames it, fsyncs the parent
directory, and releases the lock. Stale lock recovery is bounded and lock
acquisition has a timeout.
Two independently constructed adapter instances prove same-key deduplication, one-winner optimistic concurrency for distinct writers, monotonic sequence allocation, and contiguous deterministic attempt ordinals. Replica append accepts only a continuation of the complete history and rejects duplicate or nonmonotonic input. The file adapter is an actual process-boundary reference store, not a claim of multi-host database, distributed-consensus, or product runtime integration.
Checkpoints, migrations, stable resume, and replay#
The portable protocol defines strict checkpoint schema versions 1 and 2. Forward migration maps legacy thread/cursor/head fields into the current stable thread ID, sequence cursor, state digest, parent checkpoint, and retention manifest form. Backward migration emits the strict V1 representation. Tests exercise V2 to V1 to V2 compatibility, preserved state/cursor/head identity, unknown-field rejection, and unsupported-version rejection.
Checkpoint creation binds materialized RUN-001 state to the exact state-bearing event cursor and digest. Resume looks up the run by stable thread ID, loads the latest checkpoint, replays later events, independently replays from origin, and fails if the results disagree. A real second Node process opens the file and recovers the checkpointed run through the same stable thread ID.
Process-death recovery matrix#
The adapter exposes six explicit conformance fault boundaries:
- after lock acquisition;
- after latest-state load;
- before temporary-file write;
- after temporary-file fsync;
- after atomic rename; and
- after parent-directory fsync.
For each boundary, the test launches a separate Node process, attempts one
queued-to-planning transition, sends the process SIGKILL at that exact point,
and opens the same store from a recovery adapter. Every case yields either the
prior queued state or the committed planning state, never a partial record and
never fabricated success. The recovered store then accepts the correct next
transition. This is exhaustive for the adapter's durable commit boundaries; it
is not a power-loss, kernel/filesystem fault, network partition, or product
pipeline crash certification.
Compaction, archive, and retention#
Compaction first creates or reuses an integrity-bound checkpoint, then moves the eligible active prefix into a strict archive segment. The segment retains the original events and hash chain, cursor/head digests, archive digest, and a manifest for parent/fork lineage, approval records, legal records, and learning attribution. Archive import validates the internal chain and refuses a retention manifest that omits event-pinned references.
readHistory merges archived and active events in the original order, and
resume after compaction must equal origin replay. The conformance test archives
all active events while proving that every event digest and every lineage,
approval, legal, and learning reference remains available. The reference file
keeps archive segments inline for auditability; it does not claim external
cold-storage lifecycle, retention deletion, encryption, or reduced total file
size.
Artifact identity#
| Artifact | SHA-256 |
|---|---|
libs/contracts/src/run-protocol/run-model.ts |
1d210cd89e78409c0a2c73804fba391e118388f6fddc183c38e6c1b6cf110f4b |
libs/contracts/src/run-protocol/durable-run-protocol.ts |
99ccbd7627b651eaa6e08516c60541c06798edc1fba527dbca212159a28f6568 |
libs/contracts/src/run-protocol/file-durable-run-store.ts |
cd4799321d826302459ddd65a02c6b408a93342e14df84b42ed4dd6530bb76a1 |
libs/contracts/src/run-protocol/durable-run-protocol.spec.ts |
0ae481f213895bb5bc2e4168d5c86d47098efb6f6dbd87a938f2c62de6811878 |
libs/contracts/src/run-protocol/index.ts |
cf6855e981df59dda68ae2939bf3d16ae750a299cffe7284c891a50340066f97 |
package.json |
60ccf0a12c4cc16cd76e1d4b381ee86df07198bd2443e0e003ed389f449e266e |
.github/workflows/run-protocol-contract.yml |
930c1394f7b84d9610f2681c54f8a58a26b91cfb6ab841a61204ff3339c0b3a8 |
Automated and adversarial verification#
pnpm verify:contracts:run-protocolpasses targeted ESLint with zero warnings, 33/33 RUN-001/RUN-002 Vitest cases with one worker, and contracts source typechecking. RUN-002 contributes 15 focused cases.- The RUN-002 suite proves canonical thread/event identities, strict unknown
field and digest rejection, duplicate/nonmonotonic/broken-chain rejection,
archive integrity, fresh-adapter recovery, write-before-effect ordering,
identical-request deduplication, idempotency conflict rejection,
cross-instance atomic sequence/attempt allocation, optimistic race handling,
strict replica continuation, forward/backward checkpoint migration, an actual
second-process checkpoint resume, all six
SIGKILLcommit boundaries, lossless archive/retention replay, on-disk corruption rejection, and stable repeated digests. - Existing common durability compatibility remains green at 54/54
durable-job-orchestrationandjob-durability-recoverytests. pnpm verify:contracts:content-slotremains green at 17/17, with lint and typechecking, because durable state continues to validate the RUN-001 content-slot/risk/autonomy aggregate.NX_DAEMON=false pnpm exec nx run @oshun/contracts:build --parallel=1passes both before and after the crash-matrix extension. Runtime probes verify the checkpoint/event versions, file adapter, and effect barrier through the root flat export,RunProtocolContractsnamespace, and@oshun/contracts/run-protocolsubpath.- Resource gates observed about 10–11 GiB available RAM, 30 GiB free swap, and 38 GiB free disk. Each package build ran alone at parallelism one and was polled at least every 30 seconds.
- Source pre-commit ESLint, Prettier, stub-indicator, asset locking, and V5
documentation checks passed. The prohibited broad workspace typecheck was
bypassed with
SKIP_TYPECHECK=1only after focused contracts typechecking, tests, and resource-gated builds passed.
Negative proof and task boundary#
- No V1–V9 product pipeline writes this store, no product database implements the interface, no schema migration creates a production table, and no deployed worker resumes a common run. RUN-004 owns adapters/conformance and RUN-005 owns scheduler/runtime integration.
- The reference adapter serializes writers with one shared-filesystem lock. It does not prove multi-host consensus, remote object-store atomicity, database isolation, high availability, regional failover, backup restore, encryption, key rotation, or production scale/performance.
SIGKILLproves process-death behavior on this Linux host, not sudden power loss or adversarial filesystem/kernel corruption. Fsync and atomic rename are implemented, but storage-platform durability still requires its own fault certification.- Effect intent is durable before the callback, but provider invocation, authoritative result persistence, timeout/unknown reconciliation, target ownership rules, and exactly-once user-visible behavior remain RUN-003.
- Archive compaction reduces the active replay set but keeps archived events in the same reference file. External archive placement and retention/deletion policy remain unimplemented.
- The fixture is synthetic contract/reference-adapter proof, not a production run, product publication, tool mutation, legal record, learning event, or operational recovery incident.
- No user-facing UI, mobile flow, media/model pipeline, database migration, or Unreal asset changed. Playwright/mobile/browser accessibility, provider/media/database, and Unreal execution are therefore inapplicable to RUN-002.
Publication identity#
- Initial durable-spine source commit:
b957521d6a89de13d82bf8b5095682cdc8b422e8 - Crash-recovery completion source commit:
754a4ec55c812ecffc9c244169301a435501083f - First remote-integrated tested complete source:
754a4ec55c812ecffc9c244169301a435501083f - Source-documentation alignment commit:
10011dcd67900d8c1c8f422d68cd2e358500f549 - Publication metadata commit:
a38ba5ca50954e474d2363ac396693d6aa541a99