Phase 0 Triage — Faked ML / Reasoning Stack (Task 0.1)

Ledger: AGENTIC_CONTENT_QUALITY_TODOS_2026-06-13.md § Phase 0. Date: 2026-06-13. Verifier read every file/line cited below in-session.

Method#

• Fabrication scan: grep -nE 'Math\.(sin|cos|exp|random)' over each named impl file, then read each hit in context to separate real math (DPO sigmoid, cosine-LR schedule, non-secret id suffix) from fabricated training/score curves (a Math.sin(step·k) oscillation of a step/round/progress counter with no model behind it).
• Caller scan: grep -rEl "from '@nous/training'|from '@nous/llm'" across the repo, excluding node_modules, dist, .claude/worktrees, and *.spec.ts. Then inspected each importer's exact import list.

Caller findings (who consumes these packages, non-test)#

Runtime callers of the 15 named fakes outside lib+spec: zero. The only cross-package dependency is @nous/diffusion-alignment re-exporting three request types (not values). → Fail-loud must preserve those type exports; deletion would require rewiring diffusion-alignment + training/index.ts.

Triage table#

Decision key: fail-loud = keep the typed interface, replace every fabricated metric/score with throw new NotConfiguredError(...); keep = honest today, Phase 1 wires it. Deletion was considered and rejected (see Rationale).

libs/nous/training/src/ (training metric fakers)#

libs/nous/llm/src/ (reasoning result fakers / prompt builders)#

Rationale: fail-loud over deletion#

1. CLAUDE.md is explicit: "Honest fail-loud seams … are the opposite of a stub and the correct way to represent a real-but-absent integration" and "When unsure: fail loud, or ask." A typed method that throws not_configured is the sanctioned representation.
2. Lower blast radius / reversible. Deletion = remove 15 impl + 15 spec files, strip 15 export * lines from training/index.ts, and rewire the 3 type re-exports in @nous/diffusion-alignment. Fail-loud keeps the public type surface intact, so no out-of-scope cascade into the diffusion libs.
3. Names + interfaces are reserved. Ledger 0.3/2.2/5.3 reserve these names and the *Request/*Result contracts for the real Phase 1–2/5 implementations. Keeping the contract means the real trainer/searcher drops into a defined shape rather than reinventing it.
4. The honest core survives. Input validation and pure config arithmetic (prompt/batch/token counts, real DPO sigmoid, cosine-LR schedules) are real and stay; only the fabricated metric/score emission becomes a loud throw.

Deletion remains a valid alternative if the maintainer prefers a smaller surface — it is a one-commit revert of the fail-loud husks plus the index/diffusion rewire. Recorded here so the choice is explicit, not silent.

Execution plan (drives 0.2 / 0.3)#

• 0.2 — add NotConfiguredError to @nous/training; in each training file above, make the metric-emitting runtime path throw it; delete fabricated forecast fields from "planning diagnostics"; rewrite each spec to assert the throw + the surviving honest config math. Adversarial grep Math\.(sin|cos|exp) in training/src/ must return only DPO-sigmoid / cosine-LR / real-loss hits afterward.
• 0.3 — NotConfiguredError in @nous/llm; tree-of-thought / graph-of-thought / multi-step-reasoning require a provider and throw without one; reflection-prompts keeps prompt-build, fail-louds the heuristic finding path; critique-prompts kept as-is. Specs rewritten accordingly.

0.2 completion status (2026-06-13) — scope boundary, logged not silent#

Done + verified (the RLHF / preference / alignment training family this ledger targets): rlaif, reward-modeling, dpo-training, ppo-training, constitutional-ai, diffusion-dpo-training, diffusion-curriculum-dpo, flux-dpo-training, honesty-training, harmlessness-training, helpfulness-training, ipo-training, kto-training, orpo-training, safety-training, self-play-training, rejection-sampling, preference-learning — 18 modules failed-loud. Each: Math.sin|cos|exp adversarial grep CLEAN, fabricated expected* quality forecasts removed (real config counts kept), spec asserts NotConfiguredError + not_configured code. Verified by the parent (not subagent self-report): **@nous/training tsc clean

• 645/645 tests pass; @nous/diffusion-alignment tsc clean + 3/3 pass.** The three *Requesttypes re-exported by@nous/diffusion-alignment are preserved.

Cascade caught by the parent's full-lib build (the 0.1 cross-package scan missed intra-package consumers): preference-learning composed reward-modeling and read its removed forecasts; preference-aligned-inference spec minted a FLUX artifact by running the now-failing trainer (rebuilt as a typed artifact literal — a boundary test-double; impl unchanged); diffusion-alignment spec asserted a removed forecast (repointed to a real diagnostic).

NOT done — out of this ledger's scope (the dir-wide grep is over-broad vs 0.2's named target list). The identical Math.exp(-progress·k) fake-loss / oscillation pattern also lives in the image-diffusion-distillation / base-SFT training stack: supervised-fine-tuning, consistency-model-training, lcm-lora-training, flow-matching-lcm-lora, scot-training (+ the ~20 PEFT modules that compose instruction-tuning / supervised-fine-tuning). These are a different domain (image/SFT training, the 2026-06-12 SOTA integration ledger), and failing them loud cascades through the whole PEFT stack — explicitly not the content-quality concern Phase 0 addresses. Tracked here so the remaining grep hits are explained, not silently dropped. best-of-n-sampling and iterative-refinement are Phase 0.3.

Legitimate Math.* that correctly remains in training/src: real DPO sigmoid in logsigmoid losses, cosine-LR schedules, lognormal sigma noise schedules (consistency-training-data-pipeline), cosine²/temporal-shape weighting (deepcache), and Math.random().toString(36) non-secret id suffixes.

0.3 completion status (2026-06-13)#

Done + verified. Added NotConfiguredError to @nous/llm. The reasoning engines that token-substituted templates with no model now require a real LLM provider and throw without one: tree-of-thought.buildTree, graph-of-thought.buildGraph, multi-step-reasoning.reason (all fabrication helpers deleted; exported types/constructor/getStats preserved for the real Phase 1–2 impl). critique-prompts and reflection-prompts keep their honest prompt builders and model-output parsers (buildPrompt, parseCritique, parseReflection) but fail-loud the heuristic finding paths (critiqueResponse, analyzeDraft) that invented severity/scoreImpact judgments from word-counts/regex with no model. In @nous/training, best-of-n-sampling (scored candidates by id string length) and iterative-refinement (exp/sin gain curve) fail loud like the rlaif family. Verified by the parent: @nous/llm tsc clean + 423/423 tests pass; @nous/training tsc clean + 647/647 pass. The real best-of-N / self-refine for content are Phases 2.2 / 2.3 (built fresh in the content stack), not these @nous/* husks.

0.4 completion status (2026-06-13)#

Done + verified. The four libs/yemaya/agents/src/quality-assurance/ scorers each exported an LLM-judge provider interface that was never wired (declaration-only — the illusion that LLM judging exists) and fell back to a fabricated quality default when the real path was absent:

• narrative-quality-benchmark aggregated caller-supplied dimensionScores but folded a fabricated ?? 50 for every missing dimension into the weighted "quality" (7 sites, incl. the subject↔reference comparison the subagent additionally found). Now aggregates only over present dimensions with renormalized weights, exposes scoredDimensions/dimensionCoverage, and throws honestly when zero dimensions are scored (no external consumers).
• dialogue-naturalness-scorer.parseLLMScoreResponse fabricated 50/0 when the real LLM response was unparseable; now returns { reason } and leaves the score undefined so the rule-based score is left untouched. evaluateWithLLM already fails loud (throws) without a handler.
• ending-satisfaction-predictor.predictSatisfaction folded 0.5 pacing/conformity/surprise into overallScore when no ending was set; now fails loud (guard) before any analysis (genuine extra violation the subagent found beyond the dead interface).
• mystery-fairness-validator: adversarial scan confirmed the remaining 50s are legitimate per-subgenre config thresholds and the other defaults are domain-constant lookups / vacuous-truth divide-by-zero guards — no fabricated quality default. Only the dead interface was annotated.

All three dead provider interfaces are kept and annotated Reserved for Phase 1 (content-quality-judge) wiring — NOT yet connected. Verified by the parent: yemaya/agents tsc clean; 222/222 quality-assurance tests pass.

Phase 0 net result#

The faked ML/reasoning/judge stack no longer emits a single fabricated metric, score, reasoning trace, or quality judgment when its real backend is absent — it fails loud (NotConfiguredError / throw) or reports honest absence ({ scored:false } / undefined + reason). ~30 modules across @nous/training, @nous/llm, and yemaya/agents remediated and individually test-verified. The names + typed interfaces are preserved for the real Phase 1–2 implementations. (Out of scope, logged above: the image-diffusion/SFT distillation training simulators in @nous/training share the pattern but belong to the SOTA ledger.)