Eve library

LEARN-001 evidence pack — outcome-grounded improvement loop

Evidence

Revision 1

On this page

Status: repository-complete / closed. The shared contract and Agentic Studio runtime now implement the complete evidence-to-improvement lifecycle, including exact attribution, hard-case sampling, versioned diagnosis and proposals, non-compensatory evaluation, human approval, staged promotion, automatic rollback, and dependent-artifact recertification. This closure does not claim a live telemetry deployment or a promoted production proposal.

Work-package charter#

  • Task ID: LEARN-001
  • Owner: V1-V9 evaluation governance, Agentic Studio feedback/rollout, and creative-autonomy maintainers; implementation owner: Elikem Agudogo
  • Accountable product: shared V1–V9 autonomous-content learning and evaluation plane, with Agentic Studio as the invoked lifecycle runtime
  • Priority: P0-G/P1
  • Target milestone: 2026-07-15 SOTA gap-closure program
  • Risk tier: high human-authority, policy/autonomy promotion, regression, manipulation, and dependent-artifact release risk
  • Dependencies: canonical evaluation and run/artifact attribution contracts; Agentic Studio feedback and paired champion/challenger evidence; creative-autonomy preference learning; real signal feeds, seven-dimension evaluators, human approval, rollout monitoring, and artifact recertification for production use
  • Affected schemas and content slots: canonical learning-signal, diagnosis/proposal, evaluation, approval, rollout, promotion-event, dependent-artifact, and engagement-governance contracts plus Agentic Studio snapshot version 1 and feedback adapter. Any V1–V9 slot can be referenced by exact ID, but this work package changed no prompt, model output, rendered content, media byte, or released slot itself.
  • Linked task IDs: LEARN-001, LEARN-002, EVAL-001, and EVAL-004
  • Audit finding: creative autonomy already learned from ratings, edits, rejections, and vetoes; Agentic Studio already supplied held-out gold sets and paired champion/challenger statistics. No canonical layer joined those pieces into an exactly attributed, human-governed, regression-safe system-change lifecycle.
  • Repository objective: make improvement proposals traceable to outcomes, test them on every required quality/operational axis, prevent unsafe direct engagement optimization, and make rollout history replay-verifiable.
  • Rollout strategy: additive canonical contracts and an exported, snapshot-safe Agentic Studio store/adaptor. Production composition must supply real signals, evaluators, approvals, monitoring, and recertification evidence.
  • Rollback strategy: revert source commit b1210f90706db87b94c82b0f1b2f78c6a4d629b2.
  • Evidence-pack location: this file.

Closure matrix#

Ledger requirement Closure proof
Collect explicit preference, edits, replacements, vetoes, gate failures, complaints, corrections, task outcomes, and approved contextual engagement LearningSignalKindSchema enumerates all nine kinds. OutcomeLearningLoopStore.collectSignal parses, deduplicates, snapshots, and restores them. The production feedback adapter converts existing Agentic Studio feedback without inventing diagnostic scores.
Attribute every signal to slot, run, step, artifact, prompt, model, provider, rubric, policy, source, intervention, and cohort Every signal requires every named reference plus exact artifact and prompt SHA-256. The feedback adapter rejects run or artifact substitution. Missing or blank attribution fails schema admission.
Sample hard, novel, high-impact, high-disagreement, drifted, and failed trajectories selectOutcomeLearningSignals derives all six typed reasons from a versioned policy, deduplicates the population, orders deterministically by reason count/impact/ID, applies a bounded cap, and fails loud when nothing qualifies.
Produce versioned natural-language diagnoses and proposed prompt, rubric, tool, routing, policy, or workflow changes Diagnoses require versions, narratives, evidence-bound findings, and exact selected-signal coverage. Proposals cover the six requested change types plus autonomy, require base/candidate/diff/rollback identities, and are versioned contiguously by the runtime.
Evaluate held-out capability, regression, safety, reliability, cost, latency, and subgroup suites Every offline and rollout evaluation carries threshold-derived probes for all seven dimensions. The executable runner requires all seven providers. Missing, duplicate, failed, or substituted candidate evidence blocks advancement; contamination-control and held-out suite evidence are mandatory.
Require human approval for policy/rubric/autonomy changes Those proposal kinds cannot even parse with approval disabled. Advancement requires a signed, exact-version human decision; missing, rejected, machine-authored, or substituted approval blocks.
Promote through offline, shadow, canary, and champion/challenger; automatically roll back on regression The only forward path is proposed → offline → shadow → canary → champion_challenger → production. Champion/challenger evidence must bind the candidate, contain at least 100 samples, report p≤0.05, and carry a ready verdict. The adapter accepts only Agentic Studio's paired McNemar challenger-ready report. Any failed monitored probe derives regressionDetected and automatically transitions an active rollout to rolled_back.
Invalidate and recertify dependent artifacts Production promotion invalidates every exact affected-artifact reference. Release readiness remains false until each artifact is recertified against the exact proposal version with evidence. Rollback restores the prior-version artifact state.
Never optimize raw clicks/retention directly Primary objectives are restricted to explicit outcome, preference, safety-adjusted quality, or task success. Clicks/retention can appear only as diagnostic engagement metrics. Contextual engagement signals require approved use plus user-intent, safety, explicit-outcome, and preference evidence.

Canonical evidence and attribution plane#

LearningSignal is the unit of learning authority. It retains the observed outcome, human-readable summary, evidence references, hard/failure flags, and novelty/impact/disagreement/drift scores. Its attribution binds the signal to:

  • slot, run, and step;
  • artifact reference and SHA-256;
  • prompt reference and SHA-256;
  • model and provider;
  • rubric and policy;
  • source references;
  • intervention; and
  • cohort.

There is no optional “best effort” attribution path. An absent intervention is represented explicitly (for example, intervention:none) rather than silently losing the dimension. Existing Agentic Studio feedback becomes an explicit_preference signal only when its run and output bundle match the supplied canonical attribution. Novelty, impact, disagreement, and drift scores remain caller-supplied evidence; the adapter does not fabricate them from a star rating.

Contextual clicks, retention, completion, dwell time, and return use have a separate constrained envelope. They are admitted only with an approved-use flag, user-intent reference, and non-empty safety, explicit-outcome, and preference evidence. The proposal objective schema deliberately has no raw click or retention primary metric.

Diagnosis, proposal, and non-compensatory evaluation#

Sampling uses versioned thresholds and records the entire population plus every selected reason. A diagnosis must account for every and only the selected hard cases, provide a natural-language narrative, and cite its signal evidence from each finding. The runtime requires diagnosis versions to be contiguous from 1.

Change proposals bind a diagnosis version, target, base and candidate versions, diff, objective constraints, affected artifacts, and distinct rollback target. Proposal versions are also contiguous. Policy, rubric, and autonomy proposals are structurally human-approval-required.

runLearningProposalEvaluation requires a runner for all seven dimensions:

  • held-out capability;
  • regression;
  • safety;
  • reliability;
  • cost;
  • latency; and
  • subgroup outcomes.

Each probe derives pass/fail from its score, threshold, and direction. A high mean cannot compensate for one failing axis. The evaluation binds the exact proposal/candidate, held-out suite, and contamination-control evidence.

Staged promotion, rollback, and recertification#

Every rollout stage retains seven real probes and monitoring evidence in the snapshot. Champion/challenger and production stages additionally retain the baseline/candidate comparison, sample size, p-value, verdict, and evidence. The Agentic Studio adapter maps only its paired McNemar challenger-ready report; an inconclusive, worse, undersized, or non-significant report fails loud.

Promotion events and their stage-evidence records form a one-to-one contiguous history. Snapshot restore recomputes all invariants: exact proposal/candidate, one-stage transitions, automatic-system rollback identity, stage/evidence agreement, and final stage. Removing stage evidence or rewriting a proposal, diagnosis, event, or lifecycle stage cannot manufacture a production state.

A failed monitored probe derives regression automatically. An active candidate then rolls back without depending on a caller-supplied manual transition. Only production promotion invalidates dependent artifacts, and release readiness stays false until every exact artifact is recertified with evidence for the promoted proposal version.

Adversarial fixture matrix#

Fixture Result
Signal omits provider or another attribution field Admission fails.
Feedback is rebound to another run or artifact Adapter rejects substitution.
Raw retention is supplied as the primary objective Proposal schema rejects it.
Contextual engagement lacks intent/safety/outcome/preference evidence Signal schema rejects it.
Diagnosis omits one sampled hard case Runtime rejects non-exact sample coverage.
Proposal skips a diagnosis or version Runtime rejects unknown/non-contiguous lineage.
One evaluation runner/probe is missing Evaluation fails loud or blocks.
Safety fails while other six dimensions pass Proposal remains ineligible.
Policy/rubric/autonomy proposal lacks signed human approval Offline advancement blocks.
Caller jumps directly to canary Transition rejects the skipped stages.
Champion evidence is inconclusive, undersized, non-significant, or names another candidate Adapter/schema rejects promotion evidence.
A monitored stage probe regresses Active rollout automatically transitions to rolled_back.
Snapshot loses stage evidence or substitutes the diagnosis Restore rejects the tampered snapshot.
Production artifact is recertified against another proposal version Recertification rejects the substitution.

Automated verification#

Current admitted refresh#

The current refresh ran against repository commit 9ad2678676d3c0800b9ac90bb37bb0fa1bc262a6 with lockfile SHA-256 08b055786e9b8a6104d5313601d7eef3205dedcf395066fd9a8264c9859d690f on host app-dev-server, Ubuntu 24.04.4 LTS, Linux 6.8.0-134-generic x86_64. The relevant toolchain was Node.js v24.15.0, pnpm 10.25.0, TypeScript 6.0.3, and Vitest 4.1.7.

No live model/provider, production telemetry, customer signal, human approval, canary, production candidate, or release artifact was used. Fixture identities and in-memory lifecycle transitions prove the repository authority boundary; they are not promotion evidence.

UTC interval Exit Exact command and result
2026-07-29T11:47:49.626Z–2026-07-29T11:47:50.693Z 0 pnpm exec vitest run --config libs/contracts/vitest.config.ts libs/contracts/src/common/outcome-learning-loop.test.ts --maxWorkers=1 — canonical signal, evaluation, promotion, rollback, and restore tests passed 9/9
2026-07-29T11:47:56.714Z–2026-07-29T11:48:00.448Z 0 pnpm --dir libs/oshun/agentic-studio exec vitest run src/feedback/outcome-learning-loop.test.ts --config vitest.config.ts --maxWorkers=1 — Studio runtime/adaptor tests passed 6/6
2026-07-29T11:48:06.293Z–2026-07-29T11:48:14.357Z 0 pnpm --dir libs/oshun/creative-autonomy exec vitest run src/steering/steering.test.ts src/steering/edit-diff.test.ts --config vitest.config.ts --maxWorkers=1 — preference-learning/edit tests passed 34/34
2026-07-29T11:48:22.258Z–2026-07-29T11:48:38.148Z 0 pnpm exec tsc --noEmit -p libs/contracts/tsconfig.lib.json
2026-07-29T11:48:38.156Z–2026-07-29T11:48:53.169Z 0 pnpm exec tsc --noEmit -p libs/oshun/agentic-studio/tsconfig.lib.json
2026-07-29T11:48:53.170Z–2026-07-29T11:49:09.585Z 0 pnpm exec tsc --noEmit -p libs/oshun/creative-autonomy/tsconfig.json

The admitted current suites passed 49/49 value assertions and all three owning typechecks:

  • Unit/schema/contract: all nine signal families, exact attribution, sampling, diagnosis/proposal lineage, seven non-compensatory evaluation dimensions, approval, stage ordering, rollback, recertification, snapshot, and contextual-engagement constraints are executable assertions.
  • Integration: the Agentic Studio adapter consumes canonical feedback and paired champion/challenger results; the creative learner exercises real preference, edit, calibration, deduplication, snapshot, and restoration behavior.
  • Adversarial/negative: missing/substituted attribution, raw-retention objectives, incomplete evaluation, missing or mismatched human approval, stage skipping, statistically ineligible champion evidence, monitored regression, tampered snapshots, and wrong-version recertification fail for their intended reasons.
  • Repository E2E: the in-memory lifecycle reaches ordered promotion, automatic rollback, invalidation, and recertification through public runtime APIs. It is not represented as a deployed learning service or real production promotion.

UI/accessibility automation, screenshots/media probes, mobile/Unreal automation, production performance/load testing, and a deployed E2E are inapplicable to this evidence-only refresh and the underlying contract/runtime change. No rendered surface, media artifact, mobile flow, Unreal asset, production store, or live network boundary changed.

Historical implementation verification#

At source publication, the owning packages passed 4,706/4,706 non-duplicated tests:

Surface Result
Canonical contracts full package 4,095/4,095
Agentic Studio full package 343/343
Creative autonomy full package 268/268

After final stage-evidence retention and lifecycle-history hardening, the exact source focused suites passed 9/9 canonical contract tests and 6/6 Agentic Studio runtime/adaptor tests. After rebasing an upstream root TypeScript configuration change, both focused suites passed again (15/15), and all three affected typechecks passed: contracts, Agentic Studio, and creative autonomy.

The contracts and Agentic Studio emitted TypeScript builds passed before and after the TypeScript-configuration rebase. Explicit ESLint over all changed production and test TypeScript passed with zero warnings. Prettier and git diff --check passed.

Representative commands:

text
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm --dir libs/oshun/agentic-studio exec vitest run --config vitest.config.ts
pnpm --dir libs/oshun/creative-autonomy exec vitest run --config vitest.config.ts
pnpm --dir libs/contracts exec vitest run src/common/outcome-learning-loop.test.ts
pnpm --dir libs/oshun/agentic-studio exec vitest run src/feedback/outcome-learning-loop.test.ts
pnpm exec tsc --noEmit -p libs/contracts/tsconfig.lib.json
pnpm exec tsc --noEmit -p libs/oshun/agentic-studio/tsconfig.lib.json
pnpm exec tsc --noEmit -p libs/oshun/creative-autonomy/tsconfig.json
pnpm exec tsc -p libs/contracts/tsconfig.lib.json
pnpm exec tsc -p libs/oshun/agentic-studio/tsconfig.lib.json
pnpm exec eslint --no-ignore --max-warnings=0 <all changed TypeScript>
pnpm exec prettier --check <all changed non-lock files>
git diff --check

Bounded verification notes#

  • Before the 2026-07-29 focused refresh, the host had 15 GiB RAM, 31 GiB swap, 11 GiB available memory, and 16 GiB free disk. An unrelated checkout's long-running Vitest worker and this checkout's Nx daemon were active, so no broad contracts/workspace matrix or production build was started; the three task-focused suites and three typechecks ran serially.
  • The host had 15 GiB RAM, 31 GiB swap, and long-lived Codex, Claude, and Nx processes. Package checks ran serially. The full contracts run was supervised at intervals under 30 seconds; available memory stayed at least 4.5 GiB, above both stop thresholds.
  • Full contracts verification emitted an existing missing src/index.js.map warning and completed successfully: 179 files and 4,095 tests.
  • The conventional creative-autonomy tsconfig.lib.json path does not exist. The package's actual tsconfig.json was used and passed before and after the upstream root-configuration rebase.
  • Two source rebases integrated unrelated V10 motion-channel work. The first changed root TypeScript configuration, so focused tests, all three typechecks, and both emitted builds were repeated. The second shared only the unrelated binary lockfile and required no redundant source verification.
  • The pre-existing pnpm-lock.yaml modification was preserved byte-for-byte through hooks and both rebases (Git blob a37431842e330b3a06b030947c25bebdc72bc2b1). Generated Cargo locks and libs/urania/ were not staged.
  • The source commit used the hook's documented SKIP_TYPECHECK=1 only after all typechecks and emitted builds passed. Hook lint, formatting, asset locking, and documentation validation passed. The initial commit scope produced a warning, so the commit was amended to the allowed studio scope before publication.
  • No customer UI, PWA, admin, mobile, Unreal, or visual behavior changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.

Artifact identity#

Artifact SHA-256
Canonical outcome learning contract 14eb2226b5058284eeab3d0615028457dd7236a5511bce3b3a6bb6d750424d47
Agentic Studio learning runtime/adaptor 328940ac2648aba510f71c3500cd0f4da5d2dbe56e5893bebd45f0d55ecf4959
Paired champion/challenger implementation 606f6b89ffc58d07d56f3a066ab012b806949357fe25dd5c832c1651ef6e2d06
Existing creative preference learner 1a1154c947899bcd7546652e5ffa9499383623c12244e8c7ccdbc783f8b69b3f

All four recorded hashes still match the admitted current source. The repository runtime snapshot is version 1; diagnoses and proposals carry exact positive, contiguous versions. Production model, provider, source, prompt, rubric, policy, intervention, and cohort identities are caller-supplied exact references and hashes rather than one hard-coded test version. This evidence refresh changes none of those versions.

Release, human-authority, blocker, and generated-status disposition#

  • The repository verdict is repository-complete / closed for LEARN-001. Production promotion remains unavailable without real signals, seven held-out/monitored dimensions, contamination evidence, staged runtime results, and dependent-artifact recertification for the exact proposal.
  • Policy, rubric, and autonomy changes structurally require a signed human decision bound to the exact proposal version. Publishing this repository contract does not itself require such a decision. No reviewer identity, signature, policy approval, canary approval, or production promotion is fabricated here.
  • Remaining external dependencies are durable deployment storage, real signal producers, diagnosis/evaluation providers, authorized human reviewers, rollout traffic and monitors, and exact artifact recertification. Fixtures cannot discharge those dependencies.
  • No generated capability/status artifact advertises a live learning loop or production promotion. The completed-work-package admission verifier derives only this checked task's evidence-pack presence and identity; no generated green deployment claim was hand-edited for this refresh.

Scope and remaining seams#

  • The runtime exposes a JSON-safe snapshot/restore boundary. Deployment composition must bind it to the owning durable store and real signal feeds; this package does not claim a live telemetry deployment.
  • Natural-language diagnoses, dimension evaluations, human approvals, rollout monitors, and recertification proofs remain provider/operator evidence. Their absence blocks progress; fixtures are never promotion authority.
  • The feedback adapter covers the existing explicit-preference stream. Edit, replacement, veto, gate-failure, complaint, correction, task-outcome, and approved contextual-engagement producers can submit the same canonical signal contract without a lossy generic feedback projection.

Publication identity#

  • Source commit: b1210f90706db87b94c82b0f1b2f78c6a4d629b2
  • Source publication tip: b1210f90706db87b94c82b0f1b2f78c6a4d629b2
  • Current-lineage equivalent source commit: 64abd65ca5899f8901a54866321da95200bcb96b (same source tree after rebase)
  • Closure evidence commit referenced by the ledger: 9f88c864fe82dfad635a40557ef3269e7f87a3b5
  • Current-lineage equivalent closure commit: 3f41dba5d6d091bf5e2e4b3c958ac429d4ec363f (same evidence tree after rebase)
  • Current refresh source commit: 9ad2678676d3c0800b9ac90bb37bb0fa1bc262a6
  • Current refresh evidence commit: this evidence-only publication commit; the immutable identity is the Git commit containing this file.