# Agentic Content **Quality** Implementation Ledger Source of truth: `AGENTIC_CONTENT_QUALITY_ANALYSIS_2026-06-13.md`. This ledger closes the **quality** gap that `AGENTIC_SOTA_IMPLEMENTATION_TODOS.md` (the 2026-06-12 integration ledger) did not address. The integration plumbing is real and done; this file is about making the content **compelling**, not merely correct/safe/provenanced. ## How to use this file (read before touching a checkbox) - The checkbox `[ ]` vs `[x]` is the **sole** source of truth. `✅`/`Done`/notes in prose mean nothing. Every `[ ]`→`[x]` requires you to have **read the specific code** for that task in the same session, built it, and run its tests — per `CLAUDE.md` § Task Verification. - `[~]` = not locally actionable (external vendor / on-engine / live operation). Never silently convert `[~]` or `[ ]` to `[x]`. - Phase tags: **[P1]** keystone / unblocks everything, **[P2]** high-value follow-on, **[P3]** scale/stretch. - **Definition of done for any quality task** (the standing acceptance bar): 1. A real model call or real algorithm — no `Math.sin`/`Math.random`/hardcoded scores faking a result (adversarial scan clean: see `CLAUDE.md`). 2. A test that asserts a **computed** value against a known-correct answer or a calibrated threshold — not shape/truthiness. 3. Provenance preserved (run id, model, prompt sha256, judge scores) end-to-end. 4. For pipeline tasks: a verified end-to-end run artifact (run id + provenance bundle), per the repo's adversarial-verification standard. - **Standing rule — quality layer before scale.** Do not run any volume-generation task in Phase 9 until Phases 1–3 land for that content type. Generating 12k single-pass canon-clamped nodes produces uniform slop, which is worse than a famine. - **Standing rule — no new agent loops, no new eval systems.** Build on `@iris/agents-core` (loop, agent-run), `@yemaya/orchestration` (pipeline-runner), and `@oshun/content-release-gates`. File a gap against them instead of forking. --- ## The gap map (what this ledger fixes, by level of abstraction) | Level | "Good" means | Today | Phase that fixes it | | ------------------------- | ----------------------------- | ----------------------------------------- | ---------------------------------------- | | Token / phrase | no slop, fresh diction | nobody (Neith pulls from fixed pools) | P6 (slop), P7 (Neith) | | Sentence / voice | natural, distinctive | stats only; LLM naturalness off | P1 (judge), P6 (distinctiveness→quality) | | Passage / prose | pacing, imagery, beats land | dead literary-critic rubrics | P0 (revive), P1, P2 | | Artifact (quest/scene) | coherent, on-spec, canon-safe | ✅ covered (lore-consistency, validators) | already good — keep | | Arc (10k+ words / season) | long-range coherence, payoff | structural curve only | P4 (multi-agent long-form) | | Corpus (whole catalog) | diversity, not 12k clones | nobody | P6 | | Player-experienced | fun / moving in play | nobody (post-hoc telemetry only) | P8 (player-proxy evals) | --- ## Phase 0 — Honesty remediation (do FIRST; restores trust + removes the illusion) The `libs/nous/training/*` and `libs/nous/llm/*` "advanced" stack fabricates its outputs and is a `CLAUDE.md` stub violation. Worse, the filenames create the illusion that preference learning / inference-time search already exist, which will cause under-investment in the real thing. Either delete or make fail-loud. - [x] **0.1 [P1] Inventory & triage the faked ML stack.** Enumerate every file in `libs/nous/training/src/` (`rlaif.ts`, `reward-modeling.ts`, `best-of-n-sampling.ts`, `dpo-training.ts`, `ppo-training.ts`, `constitutional-ai.ts`, `diffusion-dpo-training.ts`, `diffusion-curriculum-dpo.ts`, `flux-dpo-training.ts`) and `libs/nous/llm/` (`tree-of-thought.ts`, `graph-of-thought.ts`, `critique-prompts.ts`, `reflection-prompts.ts`, `iterative-refinement.ts`, `multi-step-reasoning.ts`). For each, record: real algorithm vs fabricated (`Math.sin`/`Math.exp`/hardcoded), any non-test caller, and decision (delete / fail-loud / keep-and-fix). Output a triage table in the PR. Acceptance: every file has a decision with evidence (file:line of the fake or the absence of callers). - [x] **0.2 [P1] Remove or fail-loud the fabricated training jobs.** For `rlaif.ts`, `reward-modeling.ts`, `dpo-training.ts`, `ppo-training.ts`, `constitutional-ai.ts`, diffusion variants: delete the modules **or** replace the fabricated metric derivations (`deriveRoundMetrics`, `Math.sin(step*…)` oscillations, `Math.exp(...)` perplexity) with a `throw new NotConfiguredError('no training backend wired')`. No module may return a number that looks like a trained metric without a trainer behind it. Acceptance: adversarial grep for `Math.sin|Math.cos|Math.exp` in `libs/nous/training/src/` returns only legitimate math (e.g. real sigmoid in a real loss), and no test asserts the shape of a fabricated curve. - [x] **0.3 [P1] Remove or fail-loud the faked search/reasoning.** `best-of-n-sampling.ts` (hardcoded candidate strings + `Math.sin` scores), `tree-of-thought.ts` / `graph-of-thought.ts` (hardcoded thought templates, no LLM), `iterative-refinement.ts` / `reflection-prompts.ts` / `critique-prompts.ts` (rubric never sent to a model), `multi-step-reasoning.ts`: delete, or convert to interfaces that REQUIRE a real provider and throw without one. These names are reserved for the real implementations in Phases 1–2 — do not let a fake squat on them. - [x] **0.4 [P1] Kill or wire the dead literary-critic LLM judges.** In `libs/yemaya/agents/src/quality-assurance/`: `narrative-quality-benchmark.ts` (`NarrativeBenchmarkLLMProvider` ~:620, rubric prompt ~:2202), `ending-satisfaction-predictor.ts` (`EndingSatisfactionLLMProvider` ~:601), `mystery-fairness-validator.ts` (`MysteryFairnessLLMProvider` ~:549), `dialogue-naturalness-scorer.ts` (`enableLLMEvaluation:false`, `analyzeExchange` never calls the handler). Either DELETE the dead provider interfaces, or — preferably — keep them and let Phase 1 wire them. Until wired, the scorer must NOT return a `?? 50` fallback that masquerades as a quality score; it must return `{ scored:false, reason:'no judge wired' }`. Acceptance: no quality scorer emits a fabricated default when its real path is unavailable. - [x] **0.5 [P2] Honest status note in the ledgers.** Add a note to `AGENTIC_SOTA_IMPLEMENTATION_TODOS.md` and root `TODOS.md` Phase 26 that "framework complete" ≠ "quality verified," pointing at this ledger. Mark any `[x]` that attested to best-of-N / preference learning / reward models as over-marked, reverting to `[ ]` with a one-line reason. - [x] **0.6 [P2] Remove the dead inference-time-scaling accounting.** `libs/iris/reasoning-thinking` `ThinkingBudgetManager` and V6 `moirai-kernel` `tier_budget_tokens` compute budgets that never reach a model; either wire them to a real `budget_tokens` call (Phase 7.6) or mark them clearly as accounting-only and not a capability. --- ## Phase 1 — The taste signal (Q1, the keystone — everything gates on this) Creative quality has **no ground-truth verifier**. You approximate it with a calibrated LLM-judge panel per content type, calibrated against your own human gold sets, with bias mitigation. This becomes a real `eval` gate via `gateFromEvalScore` in `@oshun/content-release-gates`. - [x] **1.1 [P1] Define content-type rubrics.** New lib `libs/shared/content-quality-judge/`. Author explicit, versioned rubrics per content type: **prose** (imagery, voice, pacing, emotional beat, originality), **dialogue** (naturalness, character distinctiveness, subtext, intent clarity), **quest** (hook, stakes, agency, payoff, structural validity), **commentary** (energy, accuracy-to-moment, non-repetition). Each rubric: 4–7 dimensions, 0–100 anchors with worked examples. Reuse the rubric prose already written in the dead `narrative-quality-benchmark.ts` (~:2202) — port, don't reinvent. Acceptance: rubrics are typed, versioned, and unit-tested for schema. - [x] **1.2 [P1] Judge engine with position-bias mitigation.** Implement a pairwise + pointwise judge that, for pairwise comparisons, evaluates both (A,B) and (B,A) orderings and counts only consistent wins (mitigates the ~40% position-inconsistency). Pointwise scores run ≥2 samples and report mean + variance. Runs on `@iris/agents-core` provider routing (never a new provider). Acceptance: a fixed fixture where the known-better text wins under both orderings; ties on inconsistent verdicts. - [x] **1.3 [P1] Judge panel + aggregation.** Run N≥3 judge instances (vary model and/or rubric framing for diversity), aggregate with median + a disagreement flag. High disagreement routes to a human (active learning, Phase 5). Acceptance: panel aggregation tested; disagreement above threshold raises `needs_human:true`. - [x] **1.4 [P1] Slop / GPT-ism penalty.** Integrate an EQ-Bench-style slop score: maintain a versioned slop n-gram list (over-represented LLM phrases), penalize frequency, and apply length-control so verbose outputs don't win on coverage. Acceptance: a text stuffed with known slop phrases scores measurably lower than a clean rewrite of the same content. - [x] **1.5 [P1] Calibrate against human gold sets.** Build a calibration harness: feed the panel the human-rated items from `libs/euterpe/evals/.../human-eval` and any Studio accept/reject/edit records; compute judge↔human agreement (Cohen's κ / correlation). If divergence > 20–25%, recalibrate the rubric. Acceptance: a calibration report with κ per content type; CI fails if a shipped judge's κ drops below its recorded baseline (drift gate). - [x] **1.6 [P1] Expose as a real release gate.** Wire the panel into `@oshun/content-release-gates` via `gateFromEvalScore` so a content artifact can be gated on `qualityScore >= threshold` with the panel's evidence (scores + rationales + slop). Acceptance: a release-gate suite test where a low-quality artifact fails the eval gate and a high-quality one passes, both with evidence bound to the artifact content hash. - [x] **1.7 [P2] Replace the arithmetic "AAA validators" facade.** The `aaa-quality-metrics` / `aaa-readiness-calculator` / `emotional-resonance-predictor` / `tension-curve-analyzer` / `cliche-trope-detector` family currently does weighted sums over caller-supplied scores. Re-point their leaf inputs at the real judge panel (1.1–1.6) so "AAA quality" reflects a measured signal, not bookkeeping. Acceptance: at least the narrative/dialogue aggregators read judge output; none returns a `?? 50` default. - [x] **1.8 [P2] Voice distinctiveness → voice quality.** Keep the real linguistic stats in `character-voice-distinctiveness-analyzer.ts` (type-token, Jaccard, n-gram) but add a judge dimension that rates whether a distinct voice is _good_ (not just different). Acceptance: two equally-distinct voices, one well-written and one stilted, separate on the quality dimension. --- ## Phase 2 — Inference-time quality (Q2 best-of-N, Q3 self-refine) Now that a taste signal exists, spend inference compute to raise quality. This is the single largest quality jump available and reuses the real Iris loop. - [x] **2.1 [P1] Candidate generation with diversity.** Add a `generateCandidates` path to the prose generators (`libs/hathor/narrative-generation/src/generators.ts` `completeJson`; `libs/yemaya/agents/src/creative/base-creative.ts`) that produces k candidates by varying temperature/seed and/or strategy framing (not k identical samples). Provenance records all k. Acceptance: k distinct candidates with measurably different embeddings for the same prompt. - [x] **2.2 [P1] Best-of-N selection against the judge (pessimistic).** Score the k candidates with the Phase-1 panel and select the best **pessimistically** (e.g. penalize by score variance / use a lower-confidence bound) to resist reward-hacking the judge. This is the REAL `best-of-n-sampling.ts` (replaces the Phase-0-deleted fake). Acceptance: on a fixture where one candidate is clearly best, it is chosen; selection never picks a high-slop outlier the judge over-rewards. - [x] **2.3 [P1] Structured self-refine (one pass).** Add a critique→revise step: the judge panel emits multi-dimensional, specific feedback (not "make it better"); the writer revises once against that feedback. Cap iterations; stop early if the score regresses. Acceptance: revised output beats the first draft on the panel for a majority of a fixture set (target the ~20% human-pref uplift the literature reports), with a guard that aborts on regression. - [x] **2.4 [P2] Reflexion-style memory for repeated tasks.** For batch generation (e.g. many quests), carry forward a short "lessons" memory of judge critiques so later items avoid earlier mistakes. Use `libs/iris/memory` tiers; do not build a new memory store. Acceptance: later batch items show fewer repeated critique categories than earlier ones on a controlled run. - [x] **2.5 [P2] Compute-budget control per mode.** Wire candidate count and refine passes to the V1 `fast | balanced | deep | exhaustive` modes (`V1/features.md` § Cost-Quality Controls) so cost scales with the quality bar. Enforce via the Iris `budget.ts` cost caps. Acceptance: mode selection changes k and refine depth and the cost ledger reflects it. --- ## Phase 3 — Compose the quality loop into the substrate (the "one step away" gap) The Iris loop, Yemaya pipeline-runner, AgentRun governance, and release gates are real but **uncomposed** — no production consumer wires generate→score→regenerate. Close that, and make sure the stack actually runs in a service (today it is instantiated only in integration tests). - [x] **3.1 [P1] Quality-loop orchestration step.** In `libs/yemaya/orchestration/src/execution/pipeline-runner.ts`, add a first-class `generate → judge-gate → (regenerate | refine) → re-gate` loop built on existing seams: `gateFromEvalScore` (eval verdict), `onStepFailure → replace` (swap a regenerated subgraph), and `request_changes` direction injection. Cap regenerations; escalate to human on repeated failure. Acceptance: an integration test where a deliberately weak first generation fails the judge gate, triggers a regenerate, and the improved output passes — all checkpointed/resumable. - [x] **3.2 [P1] Stand the stack up in a real service.** The agentic content stack is currently reachable only from integration tests. Mount the Iris agentic loop + Yemaya runner + release gates behind a real service (extend `apps/yemaya/*` or a new `apps/oshun` content service) with an HTTP/queue entrypoint, durable `AgentRun` persistence, and the operator dashboard surface from `V1/features.md` § Operator Job Dashboard. Acceptance: a content brief submitted via the service produces a gated artifact with a persisted run id, retrievable and replayable. (`[~]` for any cloud deploy.) _Done: new `@oshun/content-service` lib + `apps/oshun/content-service` shell. `ContentPipelineService.submitBrief` runs the EXISTING §3.1 quality loop on the `@yemaya` PipelineRunner — generate (best-of-N + self-refine over an Iris-routed writer) → judge-gate (content-release-gates suite binding the measured §1 score to the prose content hash) → `createQualityLoopFailureHandler` regenerates with the judge's dimension-targeted critique injected as `reviewerDirection` → re-gate, capped — no new loop/eval. `ContentStepDispatcher` is the judge-gate dispatcher §3.1 said would be "wired where the stack is mounted in a service." `FileContentRunStore` + `FileCheckpointStore` persist the run record and runner checkpoint to disk (durable across restart; the Postgres-per-domain backend §3.2a is a swap behind the same interfaces). `createContentHttpServer` is a real Node http entrypoint (POST /v1/content/briefs, GET /runs/:id, POST /runs/:id/replay, GET /v1/operator/runs[/:id]); the app boots it over the §7.3-pinned creative writer + judge panel and fails loud (`NotConfiguredError`) with no provider. 8 tests assert the acceptance against computed values: a brief yields a passing gated artifact bound to its content hash with a persisted run id; a low first draft regenerates and clears on round two; a never-clearing brief is `blocked`; a run is retrieved by a FRESH service instance over the same dir (restart) and replayed to reproduce the artifact under a new run id; the operator dashboard lists/filters runs; and the real booted http server round-trips submit→retrieve→dashboard. Typecheck (lib + app) + 8/8 vitest green; adversarial stub scan clean. Cloud deploy stays `[~]`._ - [x] **3.2a [P1] Persist AgentRun + gold-set capture.** Back the in-memory `ReviewStore`/`CheckpointStore`/`AgentRun` event log with real persistence (Postgres per domain DB) so runs survive restart and every human accept/reject/edit is captured as a labeled record for Phase 5. Acceptance: run durability test across process restart; human decisions land in a queryable gold-set table. - [x] **3.3 [P2] Per-step budget + kill-switch wired through the quality loop.** Ensure best-of-N and refine passes respect `BudgetManager` caps and AgentRun kill switches / anomaly quarantine at call time (not just at the outer pipeline). Acceptance: a runaway refine loop is stopped by the budget cap and audited; a kill switch aborts mid-candidate-generation. - [x] **3.4 [P2] Mid-pipeline HITL surfaces the judge evidence.** The `yemaya:approve` gate UI shows the human the judge panel scores, slop report, candidate diff, and rationales — so human review is _assisted_, not blind, and the human's override becomes a high-value gold-set label. Acceptance: the approval payload includes judge evidence and the override is recorded against the artifact. --- ## Phase 4 — Long-form & multi-agent narrative (Q4) A single planner-pinned curve does not produce arc-level coherence across 10k+ words or a season. Adopt an Agents'-Room / StoryBox pattern on the Iris multi-agent machinery. - [x] **4.1 [P2] Multi-agent narrative ensemble.** Compose specialized agents on `libs/iris/agents/multi-agent/` — planner, drafter, critic, continuity/canon agent — with hand-off contracts (reuse the V1 multi-agent protocol: budget/authority envelopes, cycle prevention, depth cap). Critic uses the Phase-1 panel. Acceptance: on a multi-chapter fixture, the ensemble beats a single-pass generation on the panel's plot/character/coherence dimensions. - [x] **4.2 [P2] Cross-artifact continuity memory.** Maintain a running story-state (characters, facts, open threads) in `libs/iris/memory` (episodic + working tiers) so chapter N is consistent with chapters 1..N-1; validate against Hathor canon (`lore-consistency.ts`) each step. Acceptance: a planted continuity contradiction in a later chapter is caught and revised. - [x] **4.3 [P2] Bottom-up simulation option (StoryBox-style).** For emergent/sandbox content, add an optional character-simulation pass that generates events from agent goals, then a narrator pass that shapes them into prose. Acceptance: a simulated scenario yields a coherent narrative that references the simulated events (no invented events — fail loud if the narrator fabricates beats not in the simulation log). - [x] **4.4 [P3] Long-form coherence eval.** Add an arc-level judge that scores setup/payoff, escalation, and thread resolution across a whole work (not per passage). Acceptance: a work with an unresolved planted thread scores lower on payoff than its resolved variant. --- ## Phase 5 — Learned taste & continuous improvement (Q5; turns humans into leverage) Your reviewers' decisions are the labeled preference data a taste model needs. Today `libs/maat/agents/agent-feedback-loop-system.ts` stores RLHF pairs and trains nothing. Close the loop. - [x] **5.1 [P1] Gold-set pipeline.** Promote captured human accept/reject/edit decisions (from 3.2a) into versioned, privacy-reviewed gold sets per content type (the V1 spec already calls for this). Acceptance: a gold set with provenance, version, and a held-out split exists and is queryable. - [x] **5.2 [P2] Reward model v1 = the calibrated judge panel.** Treat the Phase-1 panel as the reward model and validate it against the held-out gold split (this is your LitBench-equivalent). Track reward-hacking signals (length, slop, formatting correlation). Acceptance: a reward-model eval report; flagged hacking signals feed the pessimistic selection in 2.2. - **5.3 [P3] Reward model v2 = trained scorer.** When gold-set volume permits, train/fine-tune a dedicated preference scorer (DPO/reward-model — the REAL version of the Phase-0-deleted fakes) on the gold sets. Requires a real training backend (`[~]` if no GPU/training infra) — fail loud, never fabricate. Acceptance: trained scorer beats the panel on held-out agreement, or the task stays `[~]` with an honest reason. _Moved (2026-09-18): the same deliverable is K.1 of `docs/audits/AGENTIC_CONTENT_GAP_CLOSURE_TODOS_2026-06-14.md`, this file's successor; one box per deliverable._ - [x] **5.4 [P2] Active learning loop.** Route low-confidence / high-disagreement (1.3) and near-threshold (1.6) items to human review; everything else auto-resolves. Track the human-touch rate over time. Acceptance: on a controlled stream, human-touch rate falls as the gold set grows while quality holds — proving the human stops being the per-artifact bottleneck. - [x] **5.5 [P2] Champion-challenger on a learned-quality metric.** Wire `champion-challenger.ts` so prompts/models/workflows are promoted on the reward-model quality score (not just a binary success bit). The z-test is already real and verified; feed it quality outcomes. Acceptance: a better generator config is promoted only after statistical readiness on the quality metric; a worse one is retired. - [x] **5.6 [P2] Quality drift detection.** Monitor quality score, slop rate, abstention, and diversity over time per agent/content type (V1 § Feedback Loops drift detection). Acceptance: an injected quality regression trips the drift alarm and can auto-quarantine the agent version. --- ## Phase 6 — Corpus-level diversity & anti-slop (Q6; the level nothing touches) "Compelling at scale" dies here: 12k quests that are individually fine but collectively identical. No mechanism touches this today. - [x] **6.1 [P1] Corpus diversity metric.** Implement embedding-space diversity over a generated batch/catalog (mean pairwise distance, mode-collapse detection, cluster coverage). Requires real embeddings (see 7.1 — Sophia's are SHA-256 hashes today). Acceptance: a batch of near-duplicate outputs scores low diversity; a varied batch scores high; computed against a hand-checked fixture. _Done: `corpusDiversity` defaults to the real lexical bag-of-words distance (documented approximation; passes the hand-checked fixture) but the distance source is now a real seam — a caller that has awaited the §7.1 Sophia CLIP embedder passes dense vectors via `options.vectors` and the metric runs in that semantic space (cosine→[0,1] distance, fail-loud on dim/length mismatch). 4 added tests prove the metric follows the injected vectors, not the lexical text (lexically-varied texts with near-identical vectors read as near-duplicates; orthogonal vectors read maximally diverse for identical text). This is the "lift onto a real embedder" §6.1/§7.1 call for._ - [x] **6.2 [P1] Diversity as a batch release gate.** Add a corpus gate to `@oshun/content-release-gates`: a batch fails if diversity < threshold or slop frequency > threshold, even if every item passes its individual quality gate. Acceptance: a homogeneous 50-item batch fails the corpus gate while its items pass individually. - [x] **6.3 [P2] Diversity pressure in generation.** Feed corpus state back into candidate generation (2.1) — penalize candidates too close to already-chosen siblings (novelty pressure / MAP-Elites-lite over a behavior descriptor like tone×structure×theme). Acceptance: generating N items with the pressure on yields measurably higher corpus diversity than with it off. - [x] **6.4 [P2] Slop list maintenance job.** A periodic job that re-derives the slop n-gram list from recent outputs vs human-authored reference text (slop-forensics style), versioned. Acceptance: the list updates and the Phase-1 slop penalty consumes the new version. --- ## Phase 7 — Grounding & model routing (Q7 + the RAG / Neith fixes) - [x] **7.1 [P1] Real embeddings for Sophia.** Replace the SHA-256 "embeddings" in `libs/sophia/embeddings/.../multimodal-embedder.ts:45` (and the string heuristic `crossEncoderScore`) with a real embedding model via `@iris/agents-core` / Nous inference. This unblocks 6.1 diversity and real retrieval. Acceptance: semantically similar texts have high cosine similarity and dissimilar ones low, against a labeled fixture; no hash-based vector remains. _Strong-verified 2026-06-13: the named targets (`multimodal-embedder.ts`, `crossEncoderScore`) are real (CLIP ViT-B/32 + all-MiniLM via `@huggingface/transformers`, 7/7 model-backed tests). An adversarial pass found a residual SHA-256 hash-as-embedding in the SIBLING `image-embedder.ts` (`generateDeterministicEmbedding`/`hashToEmbedding` + an "in a production environment this would delegate to a real CLIP model" stub comment) — dead in prod but in scope per "no hash-based vector remains." FIXED: `ImageEmbedder` now delegates to the real `ClipMultimodalBackend` (injectable; diagram label/concept text fused into the shared CLIP space), the hash funcs are deleted, and a real-CLIP integration test asserts 512-d normalized vectors that separate two real images. 297/297 sophia-embeddings tests green; adversarial scan clean — no hash-based vector remains._ - [x] **7.2 [P1] Make grounding a mandatory generation input where canon matters.** Today real retrieval→LLM binding exists only in `apps/oshun/bff` (`domain-stubs.ts:397` + `answer-synthesizer.ts`) and grounds on Nisaba. Bind Sophia/Hathor canon retrieval as a required input to prose generation (not caller-stuffed context), with `sophia.ground` returning `unsupported-claim` rather than letting the writer invent. Acceptance: a generation that would assert an ungrounded canon fact is blocked or flagged. _Strong-verified 2026-06-13: the `grounding-gate.ts` `GateDefinition` is real + unit-tested but the adversarial pass found it called only from its own spec — never wired into a generation path. WIRED: `@oshun/content-service` now mounts the grounding gate as an opt-in **required** gate (a `GroundingRetriever` + `ClaimExtractor` on the service config); the judge-gate extracts the artifact's canon claims, runs `checkGrounding`, and an ungrounded claim fails the suite — blocking the run and surfacing the unsupported claims as regeneration guidance. Opt-in so non-canon content is not forced through it. 3 end-to-end service tests (block / pass / regenerate-to-clear) green; 11/11 content-service tests._ - [x] **7.3 [P1] Route prose to the best creative model.** Stop treating model id as decorative config. Pin prose generation and judging to the best creative-writing models (per `CLAUDE.md`: Opus 4.8 / Fable 5; verify against current EQ-Bench/Mazur leaders) via the Iris model-orchestrator, with fallback chains. Acceptance: prose paths resolve to the pinned model in the run provenance; a downgrade is visible and auditable. _Strong- verified 2026-06-13: `model-routing.ts` (pinned chains, fallback, `NoAvailableModelError` fail-loud, `creativeModelProvenance` downgrade note) is real + unit-tested and the run provenance records the resolved model. The adversarial pass found the deployable shell (`apps/oshun/content-service/main.ts`) called `resolveCreativeModel` with the decorative `() => true` default and discarded the provenance. HARDENED: the shell now passes a real `isAvailable` (an ops model kill-switch via `CONTENT_SERVICE_DISABLED_MODELS`, so the fallback chain is genuinely reachable and fails loud if the whole chain is disabled) and logs the `creativeModelProvenance` note for prose + judge, so a downgrade is visible and auditable at the call site. Typecheck green._ - [x] **7.4 [P2] Replace Neith template-prose with real generation.** The `libs/neith/integration-hathor/src/content-creation/*` generators (`narrative-generator.ts` `TENSION_VOCAB`/`TRAIT_BEAT_PATTERNS`, `quest-generator.ts` token pools, `wiki-generator.ts`) emit fixed-pool prose with zero LLM. Keep them as the **deterministic planner** (structure is good), but route all surfaced _prose_ through the Phase-1/2 writer+judge stack so no fixed-pool sentence reaches a player. Acceptance: no player-facing string originates from a `TENSION_VOCAB`-style array. _Done: the player-facing prose of every fixed-pool generator now routes through the real `@oshun/content-quality-judge` `ProseSurfacer` (writer authors fresh prose from the deterministic plan; **fails loud** `FixedPoolLeakError` if it parrots a template phrase back). **Narrative**: `NarrativeProseSurfacer` re-surfaces each act over `TENSION_FIXED_POOL`, and the canonical production path (`@hathor/narrative-generation` `generateStoryBeats`) already consumes Neith's `NarrativeGenerator` only as a structural planner (`plan.acts` → LLM prose; the `TENSION_VOCAB` `description` is used solely for internal motif filtering, never shipped). **Quest**: new `QuestProseSurfacer` re-surfaces the quest title, description, each objective description, and each complication description over `QUEST_FIXED_POOL` (derived from the now-exported `QUEST_TITLE_TEMPLATES` / `OBJECTIVE_BLUEPRINTS` / `COMPLICATION_DESCRIPTIONS` + the `generateDescription` flavour fragments), with per-field provenance. A 10-type drift guard asserts the deterministic planner's prose is caught by the pool, the surfaced prose is leak-free, structure is preserved, and a parroting writer fails loud (5 tests). **Wiki**: `WikiGenerator` content is data-grounded template formatting (real TF-IDF cross-linking + ranking + fact extraction over real world data), NOT interchangeable fixed-pool slop; re-surfacing it through a free writer would risk fabricating facts (a grounding violation), so it is correctly left as the grounded data view. The deterministic generators are untouched (they remain the planner). Typecheck + 217 (judge) / 73 (neith content-creation) vitest green._ - [x] **7.5 [P2] Wire the dormant Iris RAG + Lilith svc-ai RAG.** `libs/iris/knowledge/rag` is injection-ready but unwired; `apps/lilith/svc-ai` has an uninstantiated RAG wrapper its real LLM call bypasses. Connect them to the real embedder (7.1) and a real generator. Acceptance: a retrieval-grounded answer path runs with real vectors and cites its sources. - **7.6 [P3] Inference-time thinking budget tied to quality.** Wire a real extended-thinking budget (the correct seam exists at `libs/isis/ai-providers/.../claude4-adapter.ts:471 budget_tokens` but has no caller) into deep/exhaustive modes for hard generation/judging. Acceptance: deep mode demonstrably allocates a thinking budget that reaches the model and improves the judge score on a hard fixture. _Moved (2026-09-18): the same deliverable is K.2 of `docs/audits/AGENTIC_CONTENT_GAP_CLOSURE_TODOS_2026-06-14.md`._ --- ## Phase 8 — Eval harness & benchmarks (so quality is measured, not asserted) - [x] **8.1 [P1] Internal gold benchmark per content type.** Curate a held-out set of human-rated exemplars (good and bad) per content type. This is the ground truth the judge panel calibrates against (1.5) and that gates regressions. Acceptance: a versioned benchmark with human labels and a scoring script. - [x] **8.2 [P2] Adopt public creative benchmarks as external sanity checks.** Run EQ-Bench Creative Writing v3 (incl. slop score) and a LitBench-style reward eval against your judge/reward model to detect bias and reward hacking. Acceptance: a periodic report; large divergence from public norms triggers recalibration. - [x] **8.3 [P2] Player-experienced proxy evals.** Add evals that approximate player experience pre-ship (e.g. comprehension/engagement probes, "would you keep playing" judge prompts grounded in the artifact), distinct from the post-hoc telemetry in `engagement-predictor.ts`. Acceptance: a deliberately tedious quest scores lower on the player-proxy than an engaging one. - [x] **8.4 [P2] Quality dashboards + per-level coverage report.** Surface quality, slop, diversity, and human-touch metrics per agent/content type/product on the operator dashboard, plus a coverage report against the level-of-abstraction matrix above. Acceptance: each level shows a live metric or an explicit "unmeasured" gap. --- ## Phase 9 — Per-product rollout under the quality stack (value order; quality-before-scale) Do NOT start a product's volume generation until Phases 1–3 land for its content types. Order by value, gated on the quality layer. - [x] **9.1 [P1] V5 — the content famine, done right.** Generate side-quest + dialogue content (the audited talk/escort/hunt/scavenge patterns) through best-of-N + judge gate + corpus diversity gate + human-assisted approval — **not** the current single-pass batch driver (`libs/hathor/narrative-generation/src/batch.ts`). Stage to a few hundred items, measure quality + diversity, only then scale toward 12k. Acceptance: a measured-quality, diversity-gated batch; explicit STOP if diversity or quality regresses at scale. UE-side `V5Procgen` consumption `[~]` on-box. - **9.2 [P2] V2 — fighter Side Story vertical slice.** Run one fighter's Side Story pack (Hathor §9.6 records → prose via the quality stack → Isis concept art → Bellona cook) end-to-end with judge gates. Acceptance: a playable slice in `V2/ue` with a quality-gated, provenance-bound content bundle (`[~]` for the on-engine cook). _Moved (2026-09-18): the same deliverable is H.2 of `docs/audits/AGENTIC_CONTENT_GAP_CLOSURE_TODOS_2026-06-14.md`._ - [x] **9.3 [P2] V4 — Calliope commentary quality pass.** Add a quality dimension (energy / non-repetition / moment-fit) to `libs/calliope/match-commentary` beyond the bias-review safety gate and the deterministic scorer. Acceptance: repetitive or flat commentary fails a quality gate; the Rust service consumes the gated output (`[~]` on-box). - [x] **9.3a [P2] V4 — codex/mission narrative through the stack.** Route V4 procgen mission/codex prose through the Phase-1/2 stack. Acceptance: gated artifacts feed `V4Procgen` consumption. _Done: new `@calliope/v4-narrative` (`V4NarrativeQualityPipeline`) routes V4 side-story mission briefings (quest rubric) and codex lore (prose rubric) through best-of-N → judge gate → self-refine → corpus diversity gate on the real Phase-1/2 stack. Mission/codex plans stay the deterministic structure; only the surfaced prose goes through the writer + §1 judge panel. Each item is stamped with a `ReleaseReport` bound to the prose content hash; `selectConsumableArtifacts`/`toV4ProcgenRecord` emit only cleared, non-STOP, provenance-bound records for V4Procgen, and a STOP (corpus diversity / quality-blocked regression) withholds the whole batch. 8 tests assert computed values: gate blocks <70 / clears ≥70 bound to content hash, best-of-N selects the panel's highest candidate, self-refine keeps an improving revision and discards a regressing one, the corpus gate fails a homogeneous batch every item individually passed, and the full provenance bundle (run id, model, prompt sha256, judge score). Typecheck + 8/8 vitest green; adversarial stub scan clean._ - [x] **9.4 [P2] V3 — concert/scene authoring quality.** Add quality + diversity gates to the concert authoring pipeline alongside the existing C2PA/consent gates. Acceptance: a low-quality scene draft fails before export signing. _Done: new `@oshun/v3-concert-quality` adds `scene_quality` (every scene/speech draft scored by the §1 judge panel, slop-discounted, must meet the bar) and `scene_diversity` (corpus diversity + slop across the whole concert, §6.1) as **required** `@oshun/content-release-gates` `GateDefinition`s. `buildV3ConcertExportSuite` composes them into the V3 concert export suite immediately after the existing C2PA + consent gates, so they share the report and `promote()` (export signing). 6 tests assert computed values: a low-quality scene draft (weakest 38 < 70) fails `scene_quality` while C2PA still verifies → `promote()` throws `required gates failed`; a homogeneous 8-speech concert passes every individual quality gate but fails `scene_diversity` → signing refused; a high-quality, varied concert clears and promotes with a named signoff. Kept in a sibling lib (not the buildable `saraswati-stage`, whose `rootDir` emit config forbids cross-lib source imports — same pattern as §9.3's `@calliope/match-commentary`); the export flow composes the gates exactly as it composes the external C2PA/consent gates. Typecheck + 6/6 vitest green; saraswati-stage build config untouched; adversarial scan clean._ - [x] **9.5 [P3] V6 — cognition tier quality.** Have V6's Rust services consume the shared quality stack via the cognition gateway (`libs/iris/agents/core/src/agentic/cognition-gateway.ts`) so Egbe dialogue is judged/refined, not single-pass. Acceptance: V6 dialogue passes the judge gate through the gateway (`[~]` for V6 HTTP mount). - [x] **9.6 [P3] V7 — forge AI-assist quality.** Apply judge/refine to `libs/maya/forge-assist` generated artifacts (within the platform/realm trust boundary) before they reach the sandbox. Acceptance: low-quality assists are improved or rejected with named reasons (`[~]` for V7 sandbox integration). --- ## Cross-cutting standing rules (apply from task 0.1) - Every `[x]` on a generation/quality task requires a **verified end-to-end run artifact** (run id + provenance bundle + judge scores), per the repo's adversarial-verification standard. - No fabricated metrics, ever. A judge/reward signal must come from a real model or a real algorithm; absence is reported (`{scored:false}` / `not_configured`), never faked. Run the `CLAUDE.md` adversarial grep before every commit. - No new agent loops, no new eval systems, no new memory stores — extend `@iris/agents-core`, `@yemaya/orchestration`, `@oshun/content-release-gates`, `libs/iris/memory`. - Human sign-off surfaces are part of done for every generation feature, but the human must be _assisted by_ (and _feeding_) the taste model, never the sole per-artifact quality engine. - Quality layer before scale (see top). Phase 9 volume work is blocked on Phases 1–3 for the relevant content type. ## Suggested critical path `0.1–0.4 → 1.1–1.6 → 2.1–2.3 → 3.1–3.2a → 9.1 (staged) → 5.1–5.2/5.4 → 6.1–6.2 → 7.1–7.3`, with Phase 4 (long-form) and the remaining Phase 9 products following once the keystone (1+2+3+5) is proven on V5.