Agentic Content Quality Implementation Ledger

Source of truth: AGENTIC_CONTENT_QUALITY_ANALYSIS_2026-06-13.md. This ledger closes the quality gap that AGENTIC_SOTA_IMPLEMENTATION_TODOS.md (the 2026-06-12 integration ledger) did not address. The integration plumbing is real and done; this file is about making the content compelling, not merely correct/safe/provenanced.

How to use this file (read before touching a checkbox)#

• The checkbox [ ] vs [x] is the sole source of truth. ✅/Done/notes in prose mean nothing. Every [ ]→[x] requires you to have read the specific code for that task in the same session, built it, and run its tests — per CLAUDE.md § Task Verification.
• [~] = not locally actionable (external vendor / on-engine / live operation). Never silently convert [~] or [ ] to [x].
• Phase tags: [P1] keystone / unblocks everything, [P2] high-value follow-on, [P3] scale/stretch.
• Definition of done for any quality task (the standing acceptance bar):
  1. A real model call or real algorithm — no Math.sin/Math.random/hardcoded scores faking a result (adversarial scan clean: see CLAUDE.md).
  2. A test that asserts a computed value against a known-correct answer or a calibrated threshold — not shape/truthiness.
  3. Provenance preserved (run id, model, prompt sha256, judge scores) end-to-end.
  4. For pipeline tasks: a verified end-to-end run artifact (run id + provenance bundle), per the repo's adversarial-verification standard.
• Standing rule — quality layer before scale. Do not run any volume-generation task in Phase 9 until Phases 1–3 land for that content type. Generating 12k single-pass canon-clamped nodes produces uniform slop, which is worse than a famine.
• Standing rule — no new agent loops, no new eval systems. Build on @iris/agents-core (loop, agent-run), @yemaya/orchestration (pipeline-runner), and @oshun/content-release-gates. File a gap against them instead of forking.

The gap map (what this ledger fixes, by level of abstraction)#

Phase 0 — Honesty remediation (do FIRST; restores trust + removes the illusion)#

The libs/nous/training/* and libs/nous/llm/* "advanced" stack fabricates its outputs and is a CLAUDE.md stub violation. Worse, the filenames create the illusion that preference learning / inference-time search already exist, which will cause under-investment in the real thing. Either delete or make fail-loud.

• 0.1 [P1] Inventory & triage the faked ML stack. Enumerate every file in libs/nous/training/src/ (rlaif.ts, reward-modeling.ts, best-of-n-sampling.ts, dpo-training.ts, ppo-training.ts, constitutional-ai.ts, diffusion-dpo-training.ts, diffusion-curriculum-dpo.ts, flux-dpo-training.ts) and libs/nous/llm/ (tree-of-thought.ts, graph-of-thought.ts, critique-prompts.ts, reflection-prompts.ts, iterative-refinement.ts, multi-step-reasoning.ts). For each, record: real algorithm vs fabricated (Math.sin/Math.exp/hardcoded), any non-test caller, and decision (delete / fail-loud / keep-and-fix). Output a triage table in the PR. Acceptance: every file has a decision with evidence (file:line of the fake or the absence of callers).
• 0.2 [P1] Remove or fail-loud the fabricated training jobs. For rlaif.ts, reward-modeling.ts, dpo-training.ts, ppo-training.ts, constitutional-ai.ts, diffusion variants: delete the modules or replace the fabricated metric derivations (deriveRoundMetrics, Math.sin(step*…) oscillations, Math.exp(...) perplexity) with a throw new NotConfiguredError('no training backend wired'). No module may return a number that looks like a trained metric without a trainer behind it. Acceptance: adversarial grep for Math.sin|Math.cos|Math.exp in libs/nous/training/src/ returns only legitimate math (e.g. real sigmoid in a real loss), and no test asserts the shape of a fabricated curve.
• 0.3 [P1] Remove or fail-loud the faked search/reasoning. best-of-n-sampling.ts (hardcoded candidate strings + Math.sin scores), tree-of-thought.ts / graph-of-thought.ts (hardcoded thought templates, no LLM), iterative-refinement.ts / reflection-prompts.ts / critique-prompts.ts (rubric never sent to a model), multi-step-reasoning.ts: delete, or convert to interfaces that REQUIRE a real provider and throw without one. These names are reserved for the real implementations in Phases 1–2 — do not let a fake squat on them.
• 0.4 [P1] Kill or wire the dead literary-critic LLM judges. In libs/yemaya/agents/src/quality-assurance/: narrative-quality-benchmark.ts (NarrativeBenchmarkLLMProvider ~:620, rubric prompt ~:2202), ending-satisfaction-predictor.ts (EndingSatisfactionLLMProvider ~:601), mystery-fairness-validator.ts (MysteryFairnessLLMProvider ~:549), dialogue-naturalness-scorer.ts (enableLLMEvaluation:false, analyzeExchange never calls the handler). Either DELETE the dead provider interfaces, or — preferably — keep them and let Phase 1 wire them. Until wired, the scorer must NOT return a ?? 50 fallback that masquerades as a quality score; it must return { scored:false, reason:'no judge wired' }. Acceptance: no quality scorer emits a fabricated default when its real path is unavailable.
• 0.5 [P2] Honest status note in the ledgers. Add a note to AGENTIC_SOTA_IMPLEMENTATION_TODOS.md and root TODOS.md Phase 26 that "framework complete" ≠ "quality verified," pointing at this ledger. Mark any [x] that attested to best-of-N / preference learning / reward models as over-marked, reverting to [ ] with a one-line reason.
• 0.6 [P2] Remove the dead inference-time-scaling accounting. libs/iris/reasoning-thinking ThinkingBudgetManager and V6 moirai-kernel tier_budget_tokens compute budgets that never reach a model; either wire them to a real budget_tokens call (Phase 7.6) or mark them clearly as accounting-only and not a capability.

Phase 1 — The taste signal (Q1, the keystone — everything gates on this)#

Creative quality has no ground-truth verifier. You approximate it with a calibrated LLM-judge panel per content type, calibrated against your own human gold sets, with bias mitigation. This becomes a real eval gate via gateFromEvalScore in @oshun/content-release-gates.

• 1.1 [P1] Define content-type rubrics. New lib libs/shared/content-quality-judge/. Author explicit, versioned rubrics per content type: prose (imagery, voice, pacing, emotional beat, originality), dialogue (naturalness, character distinctiveness, subtext, intent clarity), quest (hook, stakes, agency, payoff, structural validity), commentary (energy, accuracy-to-moment, non-repetition). Each rubric: 4–7 dimensions, 0–100 anchors with worked examples. Reuse the rubric prose already written in the dead narrative-quality-benchmark.ts (~:2202) — port, don't reinvent. Acceptance: rubrics are typed, versioned, and unit-tested for schema.
• 1.2 [P1] Judge engine with position-bias mitigation. Implement a pairwise + pointwise judge that, for pairwise comparisons, evaluates both (A,B) and (B,A) orderings and counts only consistent wins (mitigates the ~40% position-inconsistency). Pointwise scores run ≥2 samples and report mean + variance. Runs on @iris/agents-core provider routing (never a new provider). Acceptance: a fixed fixture where the known-better text wins under both orderings; ties on inconsistent verdicts.
• 1.3 [P1] Judge panel + aggregation. Run N≥3 judge instances (vary model and/or rubric framing for diversity), aggregate with median + a disagreement flag. High disagreement routes to a human (active learning, Phase 5). Acceptance: panel aggregation tested; disagreement above threshold raises needs_human:true.
• 1.4 [P1] Slop / GPT-ism penalty. Integrate an EQ-Bench-style slop score: maintain a versioned slop n-gram list (over-represented LLM phrases), penalize frequency, and apply length-control so verbose outputs don't win on coverage. Acceptance: a text stuffed with known slop phrases scores measurably lower than a clean rewrite of the same content.
• 1.5 [P1] Calibrate against human gold sets. Build a calibration harness: feed the panel the human-rated items from libs/euterpe/evals/.../human-eval and any Studio accept/reject/edit records; compute judge↔human agreement (Cohen's κ / correlation). If divergence > 20–25%, recalibrate the rubric. Acceptance: a calibration report with κ per content type; CI fails if a shipped judge's κ drops below its recorded baseline (drift gate).
• 1.6 [P1] Expose as a real release gate. Wire the panel into @oshun/content-release-gates via gateFromEvalScore so a content artifact can be gated on qualityScore >= threshold with the panel's evidence (scores + rationales + slop). Acceptance: a release-gate suite test where a low-quality artifact fails the eval gate and a high-quality one passes, both with evidence bound to the artifact content hash.
• 1.7 [P2] Replace the arithmetic "AAA validators" facade. The aaa-quality-metrics / aaa-readiness-calculator / emotional-resonance-predictor / tension-curve-analyzer / cliche-trope-detector family currently does weighted sums over caller-supplied scores. Re-point their leaf inputs at the real judge panel (1.1–1.6) so "AAA quality" reflects a measured signal, not bookkeeping. Acceptance: at least the narrative/dialogue aggregators read judge output; none returns a ?? 50 default.
• 1.8 [P2] Voice distinctiveness → voice quality. Keep the real linguistic stats in character-voice-distinctiveness-analyzer.ts (type-token, Jaccard, n-gram) but add a judge dimension that rates whether a distinct voice is good (not just different). Acceptance: two equally-distinct voices, one well-written and one stilted, separate on the quality dimension.

Phase 2 — Inference-time quality (Q2 best-of-N, Q3 self-refine)#

Now that a taste signal exists, spend inference compute to raise quality. This is the single largest quality jump available and reuses the real Iris loop.

• 2.1 [P1] Candidate generation with diversity. Add a generateCandidates path to the prose generators (libs/hathor/narrative-generation/src/generators.ts completeJson; libs/yemaya/agents/src/creative/base-creative.ts) that produces k candidates by varying temperature/seed and/or strategy framing (not k identical samples). Provenance records all k. Acceptance: k distinct candidates with measurably different embeddings for the same prompt.
• 2.2 [P1] Best-of-N selection against the judge (pessimistic). Score the k candidates with the Phase-1 panel and select the best pessimistically (e.g. penalize by score variance / use a lower-confidence bound) to resist reward-hacking the judge. This is the REAL best-of-n-sampling.ts (replaces the Phase-0-deleted fake). Acceptance: on a fixture where one candidate is clearly best, it is chosen; selection never picks a high-slop outlier the judge over-rewards.
• 2.3 [P1] Structured self-refine (one pass). Add a critique→revise step: the judge panel emits multi-dimensional, specific feedback (not "make it better"); the writer revises once against that feedback. Cap iterations; stop early if the score regresses. Acceptance: revised output beats the first draft on the panel for a majority of a fixture set (target the ~20% human-pref uplift the literature reports), with a guard that aborts on regression.
• 2.4 [P2] Reflexion-style memory for repeated tasks. For batch generation (e.g. many quests), carry forward a short "lessons" memory of judge critiques so later items avoid earlier mistakes. Use libs/iris/memory tiers; do not build a new memory store. Acceptance: later batch items show fewer repeated critique categories than earlier ones on a controlled run.
• 2.5 [P2] Compute-budget control per mode. Wire candidate count and refine passes to the V1 fast | balanced | deep | exhaustive modes (V1/features.md § Cost-Quality Controls) so cost scales with the quality bar. Enforce via the Iris budget.ts cost caps. Acceptance: mode selection changes k and refine depth and the cost ledger reflects it.

Phase 3 — Compose the quality loop into the substrate (the "one step away" gap)#

The Iris loop, Yemaya pipeline-runner, AgentRun governance, and release gates are real but uncomposed — no production consumer wires generate→score→regenerate. Close that, and make sure the stack actually runs in a service (today it is instantiated only in integration tests).

• 3.1 [P1] Quality-loop orchestration step. In libs/yemaya/orchestration/src/execution/pipeline-runner.ts, add a first-class generate → judge-gate → (regenerate | refine) → re-gate loop built on existing seams: gateFromEvalScore (eval verdict), onStepFailure → replace (swap a regenerated subgraph), and request_changes direction injection. Cap regenerations; escalate to human on repeated failure. Acceptance: an integration test where a deliberately weak first generation fails the judge gate, triggers a regenerate, and the improved output passes — all checkpointed/resumable.
• 3.2 [P1] Stand the stack up in a real service. The agentic content stack is currently reachable only from integration tests. Mount the Iris agentic loop + Yemaya runner + release gates behind a real service (extend apps/yemaya/* or a new apps/oshun content service) with an HTTP/queue entrypoint, durable AgentRun persistence, and the operator dashboard surface from V1/features.md § Operator Job Dashboard. Acceptance: a content brief submitted via the service produces a gated artifact with a persisted run id, retrievable and replayable. ([~] for any cloud deploy.) Done: new @oshun/content-service lib + apps/oshun/content-service shell. ContentPipelineService.submitBrief runs the EXISTING §3.1 quality loop on the @yemaya PipelineRunner — generate (best-of-N + self-refine over an Iris-routed writer) → judge-gate (content-release-gates suite binding the measured §1 score to the prose content hash) → createQualityLoopFailureHandler regenerates with the judge's dimension-targeted critique injected as reviewerDirection → re-gate, capped — no new loop/eval. ContentStepDispatcher is the judge-gate dispatcher §3.1 said would be "wired where the stack is mounted in a service." FileContentRunStore + FileCheckpointStore persist the run record and runner checkpoint to disk (durable across restart; the Postgres-per-domain backend §3.2a is a swap behind the same interfaces). createContentHttpServer is a real Node http entrypoint (POST /v1/content/briefs, GET /runs/:id, POST /runs/:id/replay, GET /v1/operator/runs[/:id]); the app boots it over the §7.3-pinned creative writer + judge panel and fails loud (NotConfiguredError) with no provider. 8 tests assert the acceptance against computed values: a brief yields a passing gated artifact bound to its content hash with a persisted run id; a low first draft regenerates and clears on round two; a never-clearing brief is blocked; a run is retrieved by a FRESH service instance over the same dir (restart) and replayed to reproduce the artifact under a new run id; the operator dashboard lists/filters runs; and the real booted http server round-trips submit→retrieve→dashboard. Typecheck (lib + app) + 8/8 vitest green; adversarial stub scan clean. Cloud deploy stays [~].
• 3.2a [P1] Persist AgentRun + gold-set capture. Back the in-memory ReviewStore/CheckpointStore/AgentRun event log with real persistence (Postgres per domain DB) so runs survive restart and every human accept/reject/edit is captured as a labeled record for Phase 5. Acceptance: run durability test across process restart; human decisions land in a queryable gold-set table.
• 3.3 [P2] Per-step budget + kill-switch wired through the quality loop. Ensure best-of-N and refine passes respect BudgetManager caps and AgentRun kill switches / anomaly quarantine at call time (not just at the outer pipeline). Acceptance: a runaway refine loop is stopped by the budget cap and audited; a kill switch aborts mid-candidate-generation.
• 3.4 [P2] Mid-pipeline HITL surfaces the judge evidence. The yemaya:approve gate UI shows the human the judge panel scores, slop report, candidate diff, and rationales — so human review is assisted, not blind, and the human's override becomes a high-value gold-set label. Acceptance: the approval payload includes judge evidence and the override is recorded against the artifact.

Phase 4 — Long-form & multi-agent narrative (Q4)#

A single planner-pinned curve does not produce arc-level coherence across 10k+ words or a season. Adopt an Agents'-Room / StoryBox pattern on the Iris multi-agent machinery.

• 4.1 [P2] Multi-agent narrative ensemble. Compose specialized agents on libs/iris/agents/multi-agent/ — planner, drafter, critic, continuity/canon agent — with hand-off contracts (reuse the V1 multi-agent protocol: budget/authority envelopes, cycle prevention, depth cap). Critic uses the Phase-1 panel. Acceptance: on a multi-chapter fixture, the ensemble beats a single-pass generation on the panel's plot/character/coherence dimensions.
• 4.2 [P2] Cross-artifact continuity memory. Maintain a running story-state (characters, facts, open threads) in libs/iris/memory (episodic + working tiers) so chapter N is consistent with chapters 1..N-1; validate against Hathor canon (lore-consistency.ts) each step. Acceptance: a planted continuity contradiction in a later chapter is caught and revised.
• 4.3 [P2] Bottom-up simulation option (StoryBox-style). For emergent/sandbox content, add an optional character-simulation pass that generates events from agent goals, then a narrator pass that shapes them into prose. Acceptance: a simulated scenario yields a coherent narrative that references the simulated events (no invented events — fail loud if the narrator fabricates beats not in the simulation log).
• 4.4 [P3] Long-form coherence eval. Add an arc-level judge that scores setup/payoff, escalation, and thread resolution across a whole work (not per passage). Acceptance: a work with an unresolved planted thread scores lower on payoff than its resolved variant.

Phase 5 — Learned taste & continuous improvement (Q5; turns humans into leverage)#

Your reviewers' decisions are the labeled preference data a taste model needs. Today libs/maat/agents/agent-feedback-loop-system.ts stores RLHF pairs and trains nothing. Close the loop.

• 5.1 [P1] Gold-set pipeline. Promote captured human accept/reject/edit decisions (from 3.2a) into versioned, privacy-reviewed gold sets per content type (the V1 spec already calls for this). Acceptance: a gold set with provenance, version, and a held-out split exists and is queryable.
• 5.2 [P2] Reward model v1 = the calibrated judge panel. Treat the Phase-1 panel as the reward model and validate it against the held-out gold split (this is your LitBench-equivalent). Track reward-hacking signals (length, slop, formatting correlation). Acceptance: a reward-model eval report; flagged hacking signals feed the pessimistic selection in 2.2.
• 5.3 [P3] Reward model v2 = trained scorer. When gold-set volume permits, train/fine-tune a dedicated preference scorer (DPO/reward-model — the REAL version of the Phase-0-deleted fakes) on the gold sets. Requires a real training backend ([~] if no GPU/training infra) — fail loud, never fabricate. Acceptance: trained scorer beats the panel on held-out agreement, or the task stays [~] with an honest reason. Moved (2026-09-18): the same deliverable is K.1 of docs/audits/AGENTIC_CONTENT_GAP_CLOSURE_TODOS_2026-06-14.md, this file's successor; one box per deliverable.
• 5.4 [P2] Active learning loop. Route low-confidence / high-disagreement (1.3) and near-threshold (1.6) items to human review; everything else auto-resolves. Track the human-touch rate over time. Acceptance: on a controlled stream, human-touch rate falls as the gold set grows while quality holds — proving the human stops being the per-artifact bottleneck.
• 5.5 [P2] Champion-challenger on a learned-quality metric. Wire champion-challenger.ts so prompts/models/workflows are promoted on the reward-model quality score (not just a binary success bit). The z-test is already real and verified; feed it quality outcomes. Acceptance: a better generator config is promoted only after statistical readiness on the quality metric; a worse one is retired.
• 5.6 [P2] Quality drift detection. Monitor quality score, slop rate, abstention, and diversity over time per agent/content type (V1 § Feedback Loops drift detection). Acceptance: an injected quality regression trips the drift alarm and can auto-quarantine the agent version.

Phase 6 — Corpus-level diversity & anti-slop (Q6; the level nothing touches)#

"Compelling at scale" dies here: 12k quests that are individually fine but collectively identical. No mechanism touches this today.

• 6.1 [P1] Corpus diversity metric. Implement embedding-space diversity over a generated batch/catalog (mean pairwise distance, mode-collapse detection, cluster coverage). Requires real embeddings (see 7.1 — Sophia's are SHA-256 hashes today). Acceptance: a batch of near-duplicate outputs scores low diversity; a varied batch scores high; computed against a hand-checked fixture. Done: corpusDiversity defaults to the real lexical bag-of-words distance (documented approximation; passes the hand-checked fixture) but the distance source is now a real seam — a caller that has awaited the §7.1 Sophia CLIP embedder passes dense vectors via options.vectors and the metric runs in that semantic space (cosine→[0,1] distance, fail-loud on dim/length mismatch). 4 added tests prove the metric follows the injected vectors, not the lexical text (lexically-varied texts with near-identical vectors read as near-duplicates; orthogonal vectors read maximally diverse for identical text). This is the "lift onto a real embedder" §6.1/§7.1 call for.
• 6.2 [P1] Diversity as a batch release gate. Add a corpus gate to @oshun/content-release-gates: a batch fails if diversity < threshold or slop frequency > threshold, even if every item passes its individual quality gate. Acceptance: a homogeneous 50-item batch fails the corpus gate while its items pass individually.
• 6.3 [P2] Diversity pressure in generation. Feed corpus state back into candidate generation (2.1) — penalize candidates too close to already-chosen siblings (novelty pressure / MAP-Elites-lite over a behavior descriptor like tone×structure×theme). Acceptance: generating N items with the pressure on yields measurably higher corpus diversity than with it off.
• 6.4 [P2] Slop list maintenance job. A periodic job that re-derives the slop n-gram list from recent outputs vs human-authored reference text (slop-forensics style), versioned. Acceptance: the list updates and the Phase-1 slop penalty consumes the new version.

Phase 7 — Grounding & model routing (Q7 + the RAG / Neith fixes)#

• 7.1 [P1] Real embeddings for Sophia. Replace the SHA-256 "embeddings" in libs/sophia/embeddings/.../multimodal-embedder.ts:45 (and the string heuristic crossEncoderScore) with a real embedding model via @iris/agents-core / Nous inference. This unblocks 6.1 diversity and real retrieval. Acceptance: semantically similar texts have high cosine similarity and dissimilar ones low, against a labeled fixture; no hash-based vector remains. Strong-verified 2026-06-13: the named targets (multimodal-embedder.ts, crossEncoderScore) are real (CLIP ViT-B/32 + all-MiniLM via @huggingface/transformers, 7/7 model-backed tests). An adversarial pass found a residual SHA-256 hash-as-embedding in the SIBLING image-embedder.ts (generateDeterministicEmbedding/hashToEmbedding + an "in a production environment this would delegate to a real CLIP model" stub comment) — dead in prod but in scope per "no hash-based vector remains." FIXED: ImageEmbedder now delegates to the real ClipMultimodalBackend (injectable; diagram label/concept text fused into the shared CLIP space), the hash funcs are deleted, and a real-CLIP integration test asserts 512-d normalized vectors that separate two real images. 297/297 sophia-embeddings tests green; adversarial scan clean — no hash-based vector remains.
• 7.2 [P1] Make grounding a mandatory generation input where canon matters. Today real retrieval→LLM binding exists only in apps/oshun/bff (domain-stubs.ts:397 + answer-synthesizer.ts) and grounds on Nisaba. Bind Sophia/Hathor canon retrieval as a required input to prose generation (not caller-stuffed context), with sophia.ground returning unsupported-claim rather than letting the writer invent. Acceptance: a generation that would assert an ungrounded canon fact is blocked or flagged. Strong-verified 2026-06-13: the grounding-gate.ts GateDefinition is real + unit-tested but the adversarial pass found it called only from its own spec — never wired into a generation path. WIRED: @oshun/content-service now mounts the grounding gate as an opt-in required gate (a GroundingRetriever + ClaimExtractor on the service config); the judge-gate extracts the artifact's canon claims, runs checkGrounding, and an ungrounded claim fails the suite — blocking the run and surfacing the unsupported claims as regeneration guidance. Opt-in so non-canon content is not forced through it. 3 end-to-end service tests (block / pass / regenerate-to-clear) green; 11/11 content-service tests.
• 7.3 [P1] Route prose to the best creative model. Stop treating model id as decorative config. Pin prose generation and judging to the best creative-writing models (per CLAUDE.md: Opus 4.8 / Fable 5; verify against current EQ-Bench/Mazur leaders) via the Iris model-orchestrator, with fallback chains. Acceptance: prose paths resolve to the pinned model in the run provenance; a downgrade is visible and auditable. Strong- verified 2026-06-13: model-routing.ts (pinned chains, fallback, NoAvailableModelError fail-loud, creativeModelProvenance downgrade note) is real + unit-tested and the run provenance records the resolved model. The adversarial pass found the deployable shell (apps/oshun/content-service/main.ts) called resolveCreativeModel with the decorative () => true default and discarded the provenance. HARDENED: the shell now passes a real isAvailable (an ops model kill-switch via CONTENT_SERVICE_DISABLED_MODELS, so the fallback chain is genuinely reachable and fails loud if the whole chain is disabled) and logs the creativeModelProvenance note for prose + judge, so a downgrade is visible and auditable at the call site. Typecheck green.
• 7.4 [P2] Replace Neith template-prose with real generation. The libs/neith/integration-hathor/src/content-creation/* generators (narrative-generator.ts TENSION_VOCAB/TRAIT_BEAT_PATTERNS, quest-generator.ts token pools, wiki-generator.ts) emit fixed-pool prose with zero LLM. Keep them as the deterministic planner (structure is good), but route all surfaced prose through the Phase-1/2 writer+judge stack so no fixed-pool sentence reaches a player. Acceptance: no player-facing string originates from a TENSION_VOCAB-style array. Done: the player-facing prose of every fixed-pool generator now routes through the real @oshun/content-quality-judge ProseSurfacer (writer authors fresh prose from the deterministic plan; fails loud FixedPoolLeakError if it parrots a template phrase back). Narrative: NarrativeProseSurfacer re-surfaces each act over TENSION_FIXED_POOL, and the canonical production path (@hathor/narrative-generation generateStoryBeats) already consumes Neith's NarrativeGenerator only as a structural planner (plan.acts → LLM prose; the TENSION_VOCAB description is used solely for internal motif filtering, never shipped). Quest: new QuestProseSurfacer re-surfaces the quest title, description, each objective description, and each complication description over QUEST_FIXED_POOL (derived from the now-exported QUEST_TITLE_TEMPLATES / OBJECTIVE_BLUEPRINTS / COMPLICATION_DESCRIPTIONS + the generateDescription flavour fragments), with per-field provenance. A 10-type drift guard asserts the deterministic planner's prose is caught by the pool, the surfaced prose is leak-free, structure is preserved, and a parroting writer fails loud (5 tests). Wiki: WikiGenerator content is data-grounded template formatting (real TF-IDF cross-linking + ranking + fact extraction over real world data), NOT interchangeable fixed-pool slop; re-surfacing it through a free writer would risk fabricating facts (a grounding violation), so it is correctly left as the grounded data view. The deterministic generators are untouched (they remain the planner). Typecheck + 217 (judge) / 73 (neith content-creation) vitest green.
• 7.5 [P2] Wire the dormant Iris RAG + Lilith svc-ai RAG. libs/iris/knowledge/rag is injection-ready but unwired; apps/lilith/svc-ai has an uninstantiated RAG wrapper its real LLM call bypasses. Connect them to the real embedder (7.1) and a real generator. Acceptance: a retrieval-grounded answer path runs with real vectors and cites its sources.
• 7.6 [P3] Inference-time thinking budget tied to quality. Wire a real extended-thinking budget (the correct seam exists at libs/isis/ai-providers/.../claude4-adapter.ts:471 budget_tokens but has no caller) into deep/exhaustive modes for hard generation/judging. Acceptance: deep mode demonstrably allocates a thinking budget that reaches the model and improves the judge score on a hard fixture. Moved (2026-09-18): the same deliverable is K.2 of docs/audits/AGENTIC_CONTENT_GAP_CLOSURE_TODOS_2026-06-14.md.

Phase 8 — Eval harness & benchmarks (so quality is measured, not asserted)#

• 8.1 [P1] Internal gold benchmark per content type. Curate a held-out set of human-rated exemplars (good and bad) per content type. This is the ground truth the judge panel calibrates against (1.5) and that gates regressions. Acceptance: a versioned benchmark with human labels and a scoring script.
• 8.2 [P2] Adopt public creative benchmarks as external sanity checks. Run EQ-Bench Creative Writing v3 (incl. slop score) and a LitBench-style reward eval against your judge/reward model to detect bias and reward hacking. Acceptance: a periodic report; large divergence from public norms triggers recalibration.
• 8.3 [P2] Player-experienced proxy evals. Add evals that approximate player experience pre-ship (e.g. comprehension/engagement probes, "would you keep playing" judge prompts grounded in the artifact), distinct from the post-hoc telemetry in engagement-predictor.ts. Acceptance: a deliberately tedious quest scores lower on the player-proxy than an engaging one.
• 8.4 [P2] Quality dashboards + per-level coverage report. Surface quality, slop, diversity, and human-touch metrics per agent/content type/product on the operator dashboard, plus a coverage report against the level-of-abstraction matrix above. Acceptance: each level shows a live metric or an explicit "unmeasured" gap.

Phase 9 — Per-product rollout under the quality stack (value order; quality-before-scale)#

Do NOT start a product's volume generation until Phases 1–3 land for its content types. Order by value, gated on the quality layer.

• 9.1 [P1] V5 — the content famine, done right. Generate side-quest + dialogue content (the audited talk/escort/hunt/scavenge patterns) through best-of-N + judge gate + corpus diversity gate + human-assisted approval — not the current single-pass batch driver (libs/hathor/narrative-generation/src/batch.ts). Stage to a few hundred items, measure quality + diversity, only then scale toward 12k. Acceptance: a measured-quality, diversity-gated batch; explicit STOP if diversity or quality regresses at scale. UE-side V5Procgen consumption [~] on-box.
• 9.2 [P2] V2 — fighter Side Story vertical slice. Run one fighter's Side Story pack (Hathor §9.6 records → prose via the quality stack → Isis concept art → Bellona cook) end-to-end with judge gates. Acceptance: a playable slice in V2/ue with a quality-gated, provenance-bound content bundle ([~] for the on-engine cook). Moved (2026-09-18): the same deliverable is H.2 of docs/audits/AGENTIC_CONTENT_GAP_CLOSURE_TODOS_2026-06-14.md.
• 9.3 [P2] V4 — Calliope commentary quality pass. Add a quality dimension (energy / non-repetition / moment-fit) to libs/calliope/match-commentary beyond the bias-review safety gate and the deterministic scorer. Acceptance: repetitive or flat commentary fails a quality gate; the Rust service consumes the gated output ([~] on-box).
• 9.3a [P2] V4 — codex/mission narrative through the stack. Route V4 procgen mission/codex prose through the Phase-1/2 stack. Acceptance: gated artifacts feed V4Procgen consumption. Done: new @calliope/v4-narrative (V4NarrativeQualityPipeline) routes V4 side-story mission briefings (quest rubric) and codex lore (prose rubric) through best-of-N → judge gate → self-refine → corpus diversity gate on the real Phase-1/2 stack. Mission/codex plans stay the deterministic structure; only the surfaced prose goes through the writer + §1 judge panel. Each item is stamped with a ReleaseReport bound to the prose content hash; selectConsumableArtifacts/toV4ProcgenRecord emit only cleared, non-STOP, provenance-bound records for V4Procgen, and a STOP (corpus diversity / quality-blocked regression) withholds the whole batch. 8 tests assert computed values: gate blocks <70 / clears ≥70 bound to content hash, best-of-N selects the panel's highest candidate, self-refine keeps an improving revision and discards a regressing one, the corpus gate fails a homogeneous batch every item individually passed, and the full provenance bundle (run id, model, prompt sha256, judge score). Typecheck + 8/8 vitest green; adversarial stub scan clean.
• 9.4 [P2] V3 — concert/scene authoring quality. Add quality + diversity gates to the concert authoring pipeline alongside the existing C2PA/consent gates. Acceptance: a low-quality scene draft fails before export signing. Done: new @oshun/v3-concert-quality adds scene_quality (every scene/speech draft scored by the §1 judge panel, slop-discounted, must meet the bar) and scene_diversity (corpus diversity + slop across the whole concert, §6.1) as required @oshun/content-release-gates GateDefinitions. buildV3ConcertExportSuite composes them into the V3 concert export suite immediately after the existing C2PA + consent gates, so they share the report and promote() (export signing). 6 tests assert computed values: a low-quality scene draft (weakest 38 < 70) fails scene_quality while C2PA still verifies → promote() throws required gates failed; a homogeneous 8-speech concert passes every individual quality gate but fails scene_diversity → signing refused; a high-quality, varied concert clears and promotes with a named signoff. Kept in a sibling lib (not the buildable saraswati-stage, whose rootDir emit config forbids cross-lib source imports — same pattern as §9.3's @calliope/match-commentary); the export flow composes the gates exactly as it composes the external C2PA/consent gates. Typecheck + 6/6 vitest green; saraswati-stage build config untouched; adversarial scan clean.
• 9.5 [P3] V6 — cognition tier quality. Have V6's Rust services consume the shared quality stack via the cognition gateway (libs/iris/agents/core/src/agentic/cognition-gateway.ts) so Egbe dialogue is judged/refined, not single-pass. Acceptance: V6 dialogue passes the judge gate through the gateway ([~] for V6 HTTP mount).
• 9.6 [P3] V7 — forge AI-assist quality. Apply judge/refine to libs/maya/forge-assist generated artifacts (within the platform/realm trust boundary) before they reach the sandbox. Acceptance: low-quality assists are improved or rejected with named reasons ([~] for V7 sandbox integration).

Cross-cutting standing rules (apply from task 0.1)#

• Every [x] on a generation/quality task requires a verified end-to-end run artifact (run id + provenance bundle + judge scores), per the repo's adversarial-verification standard.
• No fabricated metrics, ever. A judge/reward signal must come from a real model or a real algorithm; absence is reported ({scored:false} / not_configured), never faked. Run the CLAUDE.md adversarial grep before every commit.
• No new agent loops, no new eval systems, no new memory stores — extend @iris/agents-core, @yemaya/orchestration, @oshun/content-release-gates, libs/iris/memory.
• Human sign-off surfaces are part of done for every generation feature, but the human must be assisted by (and feeding) the taste model, never the sole per-artifact quality engine.
• Quality layer before scale (see top). Phase 9 volume work is blocked on Phases 1–3 for the relevant content type.

Suggested critical path#

0.1–0.4 → 1.1–1.6 → 2.1–2.3 → 3.1–3.2a → 9.1 (staged) → 5.1–5.2/5.4 → 6.1–6.2 → 7.1–7.3, with Phase 4 (long-form) and the remaining Phase 9 products following once the keystone (1+2+3+5) is proven on V5.