Eve library

LEARN-002 evidence pack — safe reflective optimization

Evidence

Revision 1

On this page

Status: repository-complete / closed. Canonical contracts and the Agentic Studio runtime now confine GEPA-like reflective search to isolated sandbox drafts, compute a predeclared quality/cost/latency/safety Pareto frontier, protect law and production control surfaces, separate training from held-out evaluation, and join an exact human-reviewed diff to signed staged promotion, monitoring, automatic rollback, and artifact recertification. This closure does not claim a live optimizer provider, deployed sandbox, production promotion, or production monitoring result.

Work-package charter#

  • Task ID: LEARN-002
  • Owner: V1-V9 evaluation governance and Agentic Studio feedback/rollout maintainers; implementation owner: Elikem Agudogo
  • Accountable product: shared V1–V9 autonomous-content learning and evaluation plane, with Agentic Studio as the invoked reflective-search coordinator
  • Priority: P0-G/P1
  • Target milestone: 2026-07-15 SOTA gap-closure program
  • Risk tier: high law/policy authority, prompt manipulation, sandbox escape, unsafe promotion, regression, and dependent-artifact release risk
  • Dependencies: LEARN-001 exact proposal/evaluation/promotion lifecycle; canonical contracts; Agentic Studio feedback and governed orchestration; creative-autonomy rejection mining; real sandbox attestation, reflector, training and held-out evaluators, human signatures, rollout monitoring, and recertification for production use
  • Affected schemas and content slots: reflective request, sandbox isolation receipt, candidate, four-axis objective/evaluation, human diff review, promotion authorization, snapshot version 1, and the bridge to LEARN-001 prompt/routing/workflow proposals. Only sandbox draft references are admitted; no production prompt, rubric, policy, law, rendered content, media byte, or released slot changed.
  • Linked task IDs: LEARN-002, LEARN-001, EVAL-001, and EVAL-004
  • Audit finding: creative autonomy already produced human-facing rejection-derived steering suggestions, and Agentic Studio already supplied paired champion/challenger evaluation plus the LEARN-001 outcome lifecycle. The repository had no isolated iterative reflector, real multidimensional Pareto selection, structural protected-target barrier, or signed bridge from a reviewed optimizer diff into that lifecycle.
  • Repository objective: permit bounded reflective improvement of sandbox prompt, tool-routing, and workflow drafts without granting the optimizer authority over law, grant/release policy, production prompts, or rubrics.
  • Rollout strategy: additive canonical schemas plus an exported, snapshot-safe coordinator. Production composition must supply a real isolated executor, reflector, evaluators, signatures, monitors, and recertification evidence.
  • Rollback strategy: revert source commit 6a0d06f31676a7cb3a0dadfd968540e894769840.
  • Evidence-pack location: this file.

Closure matrix#

Ledger requirement Closure proof
Run GEPA-like or equivalent reflective optimization only in a sandbox runReflectiveOptimization performs bounded iterative reflect-and-propose search. Every run requires a signed isolation receipt fixing an ephemeral isolated_sandbox, no outbound network, production-write, or secret access, an executor image digest, filesystem scope, and isolation attestation. Candidate target or lineage escape fails loud.
Optimize a predeclared quality/cost/latency/safety Pareto frontier The request declares the objective version and hard minimum quality/safety plus maximum cost/latency before search. Selection first removes candidates outside those bounds, then computes the actual non-dominated frontier across all four axes. No eligible candidate fails the run; a dominated candidate cannot be admitted.
Prevent direct edits to non-overridable law, grant policy, release policy, production prompts, or rubrics All five protected surfaces are explicit auditable enum values, but request validation admits only sandbox prompt, tool-routing, or workflow drafts. Admission maps those three targets only to LEARN-001 prompt/routing/workflow proposals, preserves the exact base/candidate/diff lineage, and forcibly requires human approval.
Use held-out and contamination-controlled evaluations Training and held-out suite references must be distinct and contamination-control evidence is mandatory. Reflector and training-evaluator interfaces never receive the held-out identity. After admission, the exact predeclared held-out suite and contamination proof drive LEARN-001's seven non-compensatory capability, regression, safety, reliability, cost, latency, and subgroup runners.
Require human-reviewed diffs, signed promotion, canary monitoring, and automatic rollback Admission requires a human approval signed over the exact candidate diff digest. Each forward stage requires a signed human promotion authorization retained with its exact stage evidence. The ordered LEARN-001 offline → shadow → canary → champion/challenger → production lifecycle supplies monitoring; any regression uses a signed system-monitor authorization and automatic rollback. Production invalidates dependent artifacts until exact-version recertification.

Sandbox and protected-authority boundary#

The canonical request can name protected targets for audit and negative testing, but it cannot validate them as executable optimizer targets. The only admitted targets are sandbox_prompt_draft, sandbox_tool_routing_draft, and sandbox_workflow_draft. A valid isolation receipt additionally proves:

  • an ephemeral isolated sandbox;
  • no outbound network, production-write, or secret access;
  • bounded filesystem and pinned executor-image identities; and
  • signed isolation and issuance evidence.

Each candidate must preserve the exact request target, draft, and base version, use a distinct candidate version and artifact digest, retain the diff reference and digest, and occupy the next contiguous iteration. The runtime rejects target substitution, draft/base substitution, duplicate candidate identity, unchanged artifacts, and malformed iteration history.

This is a structural barrier, not an instruction to an optimizer. There is no mapping from a reflective candidate to law, grant policy, release policy, production-prompt, rubric, policy, autonomy, or generic tool-change authority.

Reflective search and Pareto selection#

The reflector receives the bounded iteration number, training suite, declared four-axis objective, and prior candidates' reflections and training metrics. It never receives the held-out suite reference. A separate training evaluator scores every proposed draft and supplies evidence. This implements a GEPA-like reflect-evaluate-history loop without representing fixture output as promotion authority.

Before the first iteration, the request fixes minimum quality and safety and maximum cost units and latency. The runtime admits only candidates inside all four hard bounds. Among those it computes non-dominance: another candidate must be no worse on every axis and strictly better on at least one to dominate. The frontier is deterministic, candidate-identity sorted, retained in the run, and recomputed during snapshot restore so serialized frontier tampering fails.

Human admission, held-out evaluation, and staged authority#

A Pareto candidate crosses out of the sandbox only through an exact signed human diff review. The review binds the run, candidate, diff reference, diff SHA-256, reviewer, decision, rationale, signature evidence, and time. The coordinator converts only an approved frontier candidate into a LEARN-001 prompt, routing, or workflow proposal and records the human approval. It cannot be caller-weakened to an approval-optional proposal.

The held-out evaluation must use the request's exact held-out suite and contamination-control proof. It invokes every LEARN-001 evaluation dimension; one missing or failing dimension blocks advancement. Promotion then follows the same contiguous monitored lifecycle established by LEARN-001. Every forward transition requires an exact proposal-version, stage-evidence-bound, signed human authorization. A failed monitored probe derives regression and requires a signed system_monitor automatic-rollback authorization. Authorization records and lifecycle stage evidence are retained one-to-one and cross-validated on restore.

Production promotion invalidates dependent artifacts. Release readiness stays false until exact proposal-version recertification evidence is recorded; rollback restores the base-version state.

Adversarial fixture matrix#

Fixture Result
Request names law, grant/release policy, production prompt, or rubric Schema rejects the optimization target.
Sandbox permits network, production writes, secrets, or persistence Isolation receipt rejects the run.
Training and held-out suite are identical or contamination proof is absent Request rejects before search.
Reflector substitutes the draft, base version, or target Runtime rejects sandbox escape.
Candidate repeats an ID, base digest, or non-contiguous iteration Runtime/schema rejects the run.
No candidate satisfies every predeclared bound Search fails loud without a frontier.
Candidate is eligible but Pareto-dominated Admission rejects it.
Review is machine-authored, rejected, unsigned, or covers another diff digest Admission rejects it.
Held-out evaluation substitutes its suite or contamination proof Evaluation rejects it.
One held-out/monitored dimension is missing or fails Advancement blocks or regression is derived.
Promotion is unsigned, machine-signed, version-substituted, or skips a stage Authorization/lifecycle rejects it.
Canary or later monitoring regresses System-monitor authorization triggers automatic rollback.
Snapshot rewrites the frontier or drops/substitutes a stage authorization Atomic restore recomputation rejects it.
Production artifact is certified against another proposal version Recertification rejects the substitution.

Automated verification#

Current admitted refresh#

The current refresh ran against repository commit 3ae745ed86ccbfe5420d0b6cda89e920a8469be3 with lockfile SHA-256 08b055786e9b8a6104d5313601d7eef3205dedcf395066fd9a8264c9859d690f on host app-dev-server, Ubuntu 24.04.4 LTS, Linux 6.8.0-134-generic x86_64. The relevant toolchain was Node.js v24.15.0, pnpm 10.25.0, TypeScript 6.0.3, and Vitest 4.1.7.

No live reflector/model/provider, deployed sandbox, secret, production prompt, customer data, human signature, canary, production candidate, or release artifact was used. Isolation receipts, signed reviews, authorizations, and candidate scores are explicit test fixtures and are never represented as live promotion evidence.

UTC interval Exit Exact command and result
2026-07-29T11:51:30.229Z–2026-07-29T11:51:31.155Z 0 pnpm exec vitest run --config libs/contracts/vitest.config.ts libs/contracts/src/common/reflective-optimization.test.ts --maxWorkers=1 — sandbox, protected-target, contamination, and Pareto contract tests passed 4/4
2026-07-29T11:51:37.424Z–2026-07-29T11:51:41.502Z 0 pnpm --dir libs/oshun/agentic-studio exec vitest run src/feedback/reflective-optimization.test.ts src/runs/governed-orchestrator.test.ts --config vitest.config.ts --maxWorkers=1 — coordinator/orchestrator tests passed 12/12
2026-07-29T11:51:49.236Z–2026-07-29T11:52:04.286Z 0 pnpm exec tsc --noEmit -p libs/contracts/tsconfig.lib.json
2026-07-29T11:52:04.288Z–2026-07-29T11:52:19.165Z 0 pnpm exec tsc --noEmit -p libs/oshun/agentic-studio/tsconfig.lib.json
2026-07-29T11:52:19.167Z–2026-07-29T11:52:33.057Z 0 pnpm exec tsc --noEmit -p libs/oshun/agentic-studio/tsconfig.spec.json

The admitted current suites passed 16/16 value assertions and all three applicable typechecks:

  • Unit/schema/contract: protected targets, isolation receipts, separate train/held-out suites, contamination evidence, exact candidate lineage, four-axis hard bounds, Pareto non-dominance, signed review, and authorization records are executable assertions.
  • Integration: the Studio coordinator runs reflective search, admits only an exact reviewed frontier candidate into LEARN-001, evaluates all seven dimensions, advances in order, rolls back on regression, and restore-validates snapshots; the governed orchestrator remains compatible.
  • Adversarial/negative: protected-target requests, weakened sandboxes, training/held-out reuse, candidate substitution, empty frontiers, dominated admission, machine/unsigned/wrong-diff reviews, held-out substitution, incomplete evaluation, unauthorized/skipped promotion, regression, tampered snapshots, and wrong-version recertification fail for their intended reasons.
  • Repository E2E: public runtime APIs traverse sandbox search through LEARN-001 admission, promotion, rollback, and recertification using explicit injected evidence. This is not represented as a deployed optimizer, sandbox, provider, or production promotion.

UI/accessibility automation, screenshots/media probes, mobile/Unreal automation, production performance/load testing, and a deployed E2E are inapplicable to this evidence-only refresh and the underlying contract/runtime change. No rendered surface, media artifact, mobile flow, Unreal asset, production store, or live network boundary changed.

Historical implementation verification#

At source publication, the owning packages passed 4,447/4,447 non-duplicated tests:

Surface Result
Canonical contracts full package 4,099/4,099
Agentic Studio full package 348/348

The exact source focused suites passed 4/4 canonical contract tests and 12/12 Agentic Studio reflective/governed-orchestrator compatibility tests after the final root-configuration rebase. Contracts library, Agentic Studio library, and Agentic Studio spec typechecks passed. Both contracts and Agentic Studio emitted TypeScript builds passed before and after that rebase. Explicit ESLint over all eight changed TypeScript files passed with zero warnings. Prettier and git diff --check passed.

Representative commands:

text
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm --dir libs/oshun/agentic-studio exec vitest run --config vitest.config.ts
pnpm --dir libs/contracts exec vitest run src/common/reflective-optimization.test.ts
pnpm --dir libs/oshun/agentic-studio exec vitest run src/feedback/reflective-optimization.test.ts src/runs/governed-orchestrator.test.ts
pnpm exec tsc --noEmit -p libs/contracts/tsconfig.lib.json
pnpm exec tsc --noEmit -p libs/oshun/agentic-studio/tsconfig.lib.json
pnpm exec tsc --noEmit -p libs/oshun/agentic-studio/tsconfig.spec.json
pnpm exec tsc -p libs/contracts/tsconfig.lib.json
pnpm exec tsc -p libs/oshun/agentic-studio/tsconfig.lib.json
pnpm exec eslint --no-ignore --max-warnings=0 <all eight changed TypeScript files>
pnpm exec prettier --check <all eight changed TypeScript files>
git diff --check

Bounded verification notes#

  • Before the 2026-07-29 focused refresh, the host had 15 GiB RAM, 31 GiB swap, 11 GiB available memory, and 16 GiB free disk. An unrelated checkout's long-running Vitest worker and this checkout's Nx daemon were active, so no broad contracts/workspace matrix or production build was started; both focused suites and the three typechecks ran serially.
  • The host had 15 GiB RAM, 31 GiB swap, and competing long-lived Codex and Nx processes. Package checks ran serially. The full contracts run was supervised at intervals below 30 seconds; available memory remained at least 4.9 GiB, above both stop thresholds.
  • Full contracts verification emitted the existing missing libs/contracts/src/index.js.map source-map warning and completed successfully: 180 files and 4,099 tests.
  • The contracts spec-wide TypeScript configuration was also probed. It remains blocked by numerous pre-existing unrelated fixture typing errors (including readonly-array incompatibilities and unused imports); the new reflective optimization test did not appear in its diagnostics. Contracts library typechecking and emitted build both passed.
  • An existing void-callback incompatibility in the directly affected Agentic Studio governed-orchestrator fixture was corrected by using a block-bodied callback. That allowed the full Agentic Studio spec typecheck to pass and is covered by the 12-test compatibility rerun.
  • An upstream rebase changed root TypeScript configuration. Focused tests, all three applicable passing typechecks, and both emitted builds were repeated on the final source.
  • The pre-existing pnpm-lock.yaml modification was preserved byte-for-byte through hooks and rebase (Git blob a37431842e330b3a06b030947c25bebdc72bc2b1). Generated Cargo locks and libs/urania/ were not staged.
  • The source commit used the hook's documented SKIP_TYPECHECK=1 only after the targeted typechecks and emitted builds passed. Hook lint, formatting, asset locking, and documentation validation passed.
  • No customer UI, PWA, admin, mobile, Unreal, or visual behavior changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.

Artifact identity#

Artifact SHA-256
Canonical reflective-optimization contract e5a26d4c4662e13c4f82cf327f50156683bc98bc458f5db5875398f2f1b44adc
Agentic Studio reflective runtime/coordinator 7264ae388f4518eda09cfaacddbce75376e22e5d9085ae7ca8a48dbee57352be
Canonical LEARN-001 outcome lifecycle 14eb2226b5058284eeab3d0615028457dd7236a5511bce3b3a6bb6d750424d47
Paired champion/challenger implementation 606f6b89ffc58d07d56f3a066ab012b806949357fe25dd5c832c1651ef6e2d06
Existing rejection-derived steering miner e86d06f6b4b0261990f2e9ba907344c5a5631fbc3ea69ae38a4a36e489a07899

All five recorded hashes still match the admitted current source. The reflective coordinator snapshot is version 1; proposals retain LEARN-001's exact positive, contiguous versions. Production sandbox image, isolation attestation, training/held-out suite, contamination proof, model/provider, prompt, rubric/policy, signature, monitor, and artifact identities are caller-supplied exact references and hashes, not hard-coded fixture authority. This refresh changes none of those versions.

Release, human-authority, blocker, and generated-status disposition#

  • The repository verdict is repository-complete / closed for LEARN-002. Production promotion remains unavailable without a real isolated executor, reflector/evaluators, signed exact-diff review, stage authorizations, monitored rollout results, and dependent-artifact recertification.
  • Admission requires a human-signed exact candidate diff; every forward stage requires signed human authorization, while regression rollback requires a signed system_monitor authorization. Publishing this repository contract does not itself require those decisions. No reviewer identity, signature, promotion, monitoring result, or production approval is fabricated here.
  • Remaining external dependencies are sandbox deployment and attestation, reflector/model execution, contamination-controlled suites, authorized human reviewers, rollout traffic/monitors, durable storage, and recertification. Fixtures cannot discharge those dependencies.
  • No generated capability/status artifact advertises a live optimizer or production promotion. The completed-work-package admission verifier derives only this checked task's evidence-pack presence and identity; no generated green deployment claim was hand-edited for this refresh.

Scope and remaining seams#

  • The runtime exposes JSON-safe snapshot/restore boundaries. Deployment must bind these to an owning durable store; this package does not claim live persistence or an active optimizer service.
  • Sandbox attestations, reflected drafts, training/held-out scores, human signatures, stage monitoring, and recertification proofs remain real provider/operator evidence. Their absence blocks advancement; fixtures are never promotion authority.
  • The protected-target barrier prevents optimizer authority over law and policy. Human-authored changes to those surfaces remain governed by their existing owning review/release paths, outside reflective optimization.

Publication identity#

  • Source commit: 6a0d06f31676a7cb3a0dadfd968540e894769840
  • Source publication tip: 6a0d06f31676a7cb3a0dadfd968540e894769840
  • Current-lineage equivalent source commit: c779806e012551a6097e2bad93cb2d62482b1e2e (same source tree after rebase)
  • Closure evidence commit referenced by the ledger: 6bb46db48e4f4394c0679745ec009ac43bda2201
  • Current-lineage equivalent closure commit: 8fa6d35af5e15c4ac79eb4a7bf5798acafecf4db (same evidence tree after rebase)
  • Current refresh source commit: 3ae745ed86ccbfe5420d0b6cda89e920a8469be3
  • Current refresh evidence commit: this evidence-only publication commit; the immutable identity is the Git commit containing this file.