# RUN-002 evidence pack — durable append, checkpoint, migration, and replay

## Work-package charter

- **Task ID:** `RUN-002`
- **Owner:** Oshun common run-protocol durability maintainer; implementation
  owner: Codex
- **Accountable product:** Shared V1–V9 autonomous-content platform
- **Priority:** P0
- **Target milestone:** 2026-07-15 SOTA gap-closure program
- **Risk tier:** Critical persistence-ordering, recovery, concurrency,
  idempotency, audit-history, and release-integrity risk
- **Dependencies:** RUN-001 strict common aggregate and deterministic attempt
  identity; existing common durable-job and V8 stage-journal concepts; AUD-003
  reproducible verification; AUD-006 generated capability truth
- **Affected schemas and runtime:** Hash-chained durable run events, V1/V2
  checkpoints, forward/backward migration, retention-bearing archive segments,
  storage interface, file-backed crash-recoverable reference adapter,
  cross-process conformance and `SIGKILL` matrix, root/namespace/subpath
  exports, and focused CI
- **Rollout strategy:** Use the portable protocol as the conformance target and
  the file adapter as the process-boundary reference implementation. Add product
  database/worker adapters under RUN-004; do not replace a product store merely
  because this reference adapter exists.
- **Rollback strategy:** Revert the durable protocol, file adapter, tests,
  exports, CI expansion, audit qualification, and ledger closure together.
  Existing product orchestration remains unchanged and therefore needs no data
  rollback for RUN-002.
- **Evidence-pack location:** This file

## Durable event and write-before-effect protocol

Every durable event carries a schema version, stable run/thread identity,
store-assigned sequence, previous-event digest, event kind, timestamp,
transition reference, idempotency key, optional effect boundary, state or
attempt-allocation payload, explicit retention references, canonical payload
digest, derived event ID, and event digest. Strict parsing recomputes state,
payload, identity, and event hashes. Replay rejects sequence gaps, broken hash
links, duplicate event/idempotency identities, time regression, changed
immutable run identity, nondeterministic attempt identity, and noncontiguous
attempt ordinals.

`persistBeforeVisibleEffect` awaits a durably appended `effect-intent` event
before invoking a tool/publication callback. A failed optimistic append never
invokes the callback. This proves the intent-before-visible-effect ordering
barrier; RUN-003 still owns provider execution receipts, unknown-outcome
reconciliation, and end-to-end effect deduplication.

## Atomic file-backed reference adapter

`FileDurableRunStore` persists a strict store snapshot and uses an atomic lock
directory as its inter-process transaction boundary. Each mutation acquires the
lock, reloads and validates the latest complete hash chain, checks the expected
head, allocates sequence or attempt ordinal inside the transaction, writes a
mode-0600 temporary file, fsyncs it, atomically renames it, fsyncs the parent
directory, and releases the lock. Stale lock recovery is bounded and lock
acquisition has a timeout.

Two independently constructed adapter instances prove same-key deduplication,
one-winner optimistic concurrency for distinct writers, monotonic sequence
allocation, and contiguous deterministic attempt ordinals. Replica append
accepts only a continuation of the complete history and rejects duplicate or
nonmonotonic input. The file adapter is an actual process-boundary reference
store, not a claim of multi-host database, distributed-consensus, or product
runtime integration.

## Checkpoints, migrations, stable resume, and replay

The portable protocol defines strict checkpoint schema versions 1 and 2. Forward
migration maps legacy thread/cursor/head fields into the current stable thread
ID, sequence cursor, state digest, parent checkpoint, and retention manifest
form. Backward migration emits the strict V1 representation. Tests exercise V2
to V1 to V2 compatibility, preserved state/cursor/head identity, unknown-field
rejection, and unsupported-version rejection.

Checkpoint creation binds materialized RUN-001 state to the exact state-bearing
event cursor and digest. Resume looks up the run by stable thread ID, loads the
latest checkpoint, replays later events, independently replays from origin, and
fails if the results disagree. A real second Node process opens the file and
recovers the checkpointed run through the same stable thread ID.

## Process-death recovery matrix

The adapter exposes six explicit conformance fault boundaries:

1. after lock acquisition;
2. after latest-state load;
3. before temporary-file write;
4. after temporary-file fsync;
5. after atomic rename; and
6. after parent-directory fsync.

For each boundary, the test launches a separate Node process, attempts one
queued-to-planning transition, sends the process `SIGKILL` at that exact point,
and opens the same store from a recovery adapter. Every case yields either the
prior queued state or the committed planning state, never a partial record and
never fabricated success. The recovered store then accepts the correct next
transition. This is exhaustive for the adapter's durable commit boundaries; it
is not a power-loss, kernel/filesystem fault, network partition, or product
pipeline crash certification.

## Compaction, archive, and retention

Compaction first creates or reuses an integrity-bound checkpoint, then moves the
eligible active prefix into a strict archive segment. The segment retains the
original events and hash chain, cursor/head digests, archive digest, and a
manifest for parent/fork lineage, approval records, legal records, and learning
attribution. Archive import validates the internal chain and refuses a retention
manifest that omits event-pinned references.

`readHistory` merges archived and active events in the original order, and
resume after compaction must equal origin replay. The conformance test archives
all active events while proving that every event digest and every lineage,
approval, legal, and learning reference remains available. The reference file
keeps archive segments inline for auditability; it does not claim external
cold-storage lifecycle, retention deletion, encryption, or reduced total file
size.

## Artifact identity

| Artifact                                                       | SHA-256                                                            |
| -------------------------------------------------------------- | ------------------------------------------------------------------ |
| `libs/contracts/src/run-protocol/run-model.ts`                 | `1d210cd89e78409c0a2c73804fba391e118388f6fddc183c38e6c1b6cf110f4b` |
| `libs/contracts/src/run-protocol/durable-run-protocol.ts`      | `99ccbd7627b651eaa6e08516c60541c06798edc1fba527dbca212159a28f6568` |
| `libs/contracts/src/run-protocol/file-durable-run-store.ts`    | `cd4799321d826302459ddd65a02c6b408a93342e14df84b42ed4dd6530bb76a1` |
| `libs/contracts/src/run-protocol/durable-run-protocol.spec.ts` | `0ae481f213895bb5bc2e4168d5c86d47098efb6f6dbd87a938f2c62de6811878` |
| `libs/contracts/src/run-protocol/index.ts`                     | `cf6855e981df59dda68ae2939bf3d16ae750a299cffe7284c891a50340066f97` |
| `package.json`                                                 | `60ccf0a12c4cc16cd76e1d4b381ee86df07198bd2443e0e003ed389f449e266e` |
| `.github/workflows/run-protocol-contract.yml`                  | `930c1394f7b84d9610f2681c54f8a58a26b91cfb6ab841a61204ff3339c0b3a8` |

## Automated and adversarial verification

- `pnpm verify:contracts:run-protocol` passes targeted ESLint with zero
  warnings, **33/33** RUN-001/RUN-002 Vitest cases with one worker, and
  contracts source typechecking. RUN-002 contributes 15 focused cases.
- The RUN-002 suite proves canonical thread/event identities, strict unknown
  field and digest rejection, duplicate/nonmonotonic/broken-chain rejection,
  archive integrity, fresh-adapter recovery, write-before-effect ordering,
  identical-request deduplication, idempotency conflict rejection,
  cross-instance atomic sequence/attempt allocation, optimistic race handling,
  strict replica continuation, forward/backward checkpoint migration, an actual
  second-process checkpoint resume, all six `SIGKILL` commit boundaries,
  lossless archive/retention replay, on-disk corruption rejection, and stable
  repeated digests.
- Existing common durability compatibility remains green at **54/54**
  `durable-job-orchestration` and `job-durability-recovery` tests.
- `pnpm verify:contracts:content-slot` remains green at **17/17**, with lint and
  typechecking, because durable state continues to validate the RUN-001
  content-slot/risk/autonomy aggregate.
- `NX_DAEMON=false pnpm exec nx run @oshun/contracts:build --parallel=1` passes
  both before and after the crash-matrix extension. Runtime probes verify the
  checkpoint/event versions, file adapter, and effect barrier through the root
  flat export, `RunProtocolContracts` namespace, and
  `@oshun/contracts/run-protocol` subpath.
- Resource gates observed about 10–11 GiB available RAM, 30 GiB free swap, and
  38 GiB free disk. Each package build ran alone at parallelism one and was
  polled at least every 30 seconds.
- Source pre-commit ESLint, Prettier, stub-indicator, asset locking, and V5
  documentation checks passed. The prohibited broad workspace typecheck was
  bypassed with `SKIP_TYPECHECK=1` only after focused contracts typechecking,
  tests, and resource-gated builds passed.

## Negative proof and task boundary

- No V1–V9 product pipeline writes this store, no product database implements
  the interface, no schema migration creates a production table, and no deployed
  worker resumes a common run. RUN-004 owns adapters/conformance and RUN-005
  owns scheduler/runtime integration.
- The reference adapter serializes writers with one shared-filesystem lock. It
  does not prove multi-host consensus, remote object-store atomicity, database
  isolation, high availability, regional failover, backup restore, encryption,
  key rotation, or production scale/performance.
- `SIGKILL` proves process-death behavior on this Linux host, not sudden power
  loss or adversarial filesystem/kernel corruption. Fsync and atomic rename are
  implemented, but storage-platform durability still requires its own fault
  certification.
- Effect intent is durable before the callback, but provider invocation,
  authoritative result persistence, timeout/unknown reconciliation, target
  ownership rules, and exactly-once user-visible behavior remain RUN-003.
- Archive compaction reduces the active replay set but keeps archived events in
  the same reference file. External archive placement and retention/deletion
  policy remain unimplemented.
- The fixture is synthetic contract/reference-adapter proof, not a production
  run, product publication, tool mutation, legal record, learning event, or
  operational recovery incident.
- No user-facing UI, mobile flow, media/model pipeline, database migration, or
  Unreal asset changed. Playwright/mobile/browser accessibility,
  provider/media/database, and Unreal execution are therefore inapplicable to
  RUN-002.

## Publication identity

- **Initial durable-spine source commit:**
  `b957521d6a89de13d82bf8b5095682cdc8b422e8`
- **Crash-recovery completion source commit:**
  `754a4ec55c812ecffc9c244169301a435501083f`
- **First remote-integrated tested complete source:**
  `754a4ec55c812ecffc9c244169301a435501083f`
- **Source-documentation alignment commit:**
  `10011dcd67900d8c1c8f422d68cd2e358500f549`
- **Publication metadata commit:** `a38ba5ca50954e474d2363ac396693d6aa541a99`
