Disciplines · Audits

V1 SOTA Audit Remediation Ledger — 2026-07-02

set through the plane snapshot (director.ts:218,

8sections7 minread
Markdown source
On this page

Source: V1_AUTONOMOUS_CONTENT_SOTA_AUDIT_2026-07-02.md. Rules: one task at a time; [x] only after the code is read, built, and its tests pass in this conversation; adversarial stub scan before every commit. Data/creds-gated items are delivered as real algorithms behind honest fail-loud seams (never fabricated results) and marked (seam).

Phase 1 — P0 verified defects#

  • H1 restart double-learning: persist/restore the director's learned-item set through the plane snapshot (director.ts:218, plane-persistence.ts:42-57) — director snapshotLearnedItemIds/restoreLearnedItemIds, snapshot field + resolved-status fallback for pre-migration snapshots; regression test asserts arms unchanged after restore+cycle (e2e suite 5/5)
  • H2 heterogeneous judge panel: resolveJudgePanelModels assigns distinct models (Opus/Sonnet/Fable) per member with degraded-diversity auditing; wired into content-service main.ts buildJudgePanel (10 routing tests)
  • H3 injection fencing on all judge prompts (judge-engine + player-proxy) + judge-red-team regression suite (cqj committed)
  • H4 job-drain double-execution race: re-check queued per job before executing (jobs-route.ts:142-159); gallery cataloging verified already idempotent (OutputGalleryAdminStore.addRecord replaces on outputId) — jobs-route suite 17/17
  • H5 bind publish-readiness gates into the autonomous editorial publisher (real inputs only — Lilith review, authoring evaluation, Isis gate ids for media, rights/provenance from the catalog)
  • M1 feedback dedupe per (user, item): stable fb-{item}-{user} id + learner-level recordedFeedbackIds set (snapshotted, restore-safe) + route 409 feedback_already_recorded; learner + route tests green
  • M2 failed-topic cooldown keyed to resolution time (max history atUnixSeconds), not submission time; discriminating 7.5d-late-rejection test added (director suite 22/22)
  • M3 replace additive calibration EMA with per-category Platt/isotonic scaling once observations ≥ N (EMA as cold-start) (learner.ts:152-190)
  • M4 gate-retry parks the attempt-0 judged champion when the retry produces nothing (break to park path); completeVariantCount now accumulates across attempts like variantCount; test added
  • M5 LLM-synthesized topics sanitized at admission via sanitizeTopic (single-line, 160-char cap); hostile-topic test added
  • M6 slop detector apostrophe/quote normalization (U+2019 etc.) (slop.ts)
  • M7 run-id collision: unique run ids, never overwrite persisted run records (content-pipeline-service.ts:95-97)
  • M8 compute ceiling + budget meter wiring on the content-service HTTP path (http-router.ts, dispatcher; cap candidateCount, validate contentType)
  • M9 grounding gate configured in deployed mount: real createBriefCanonGroundingFactory (declarative-claim extraction + lexical-support retriever over brief canon, documented approximation) wired into createDurableContentService (5 tests)
  • M10 external-benchmark honesty: stop labeling authored fixtures with real dataset identities/scores (external-benchmark.ts:152-216)
  • M11 best-draft (argmax) tracking in critique-revise.ts and @oshun/ai reflexion
  • M12 deterministic critic must not read grading references from the artifact under test (move to node.params/independent measurement) (default-critic.ts)
  • M13 orchestrator governance: pre-dispatch spend recording incl. failures/revisions, honor options.signal in the dispatch loop (router.ts, governance.ts)
  • M14 gut residual fabrication stubs in libs/shared/ai/src/prompts/ (testing.ts, advanced/index.ts) to fail-loud
  • M15 media dead-path honesty: planner advertises only domains whose outputs can actually release; stop paying for guaranteed-discard cycles (server.ts:1158-1164, adapters)
  • M16 route needs_review media into the plane's review queue instead of throwing (generation-producer.ts:52-57)
  • M17 domain-routed critics: media domains get media-capable critics, never the text-LLM-over-JSON critic (create-plane.ts:125)
  • M18 analysis release exemption must be content-based, not URL-based (release-gate.ts:132-138)
  • M19 validate executor requests (no unvalidated casts; admin raw-enqueue schema) + fix provider attribution labels (image-executor.ts:61,67)
  • M20 wire gold-set consumption: scheduled assembly → holdout → panel re-score → κ report feeding per-category calibration
  • M21 wire editorial calendar + plane planCalendar into the publish path (scheduled publishing, embargo law)
  • M22 voice-disclosure gate: synthesis-disclosure canonical gate blocks synthetic-voice audio/video lacking an audible AI-voice disclosure (backward-compat optional measurement fields); isis suite 410/410 matching the release-gate-model header contract
  • M23 McNemar paired test for champion-challenger readiness + durable editorial records/transition history

Phase 2 — LOW defects#

  • L1 pointwise variance at a fixed operating point (no temperature ladder conflation) (judge-engine.ts:220-224)
  • L2 κ/Pearson fail loud on mismatched arrays (calibration.ts)
  • L3 drift guards: n≥2 windows for Welch z; editorial drift delta===0 not "better" (drift.ts, drift-detection.ts)
  • L4 structured-output.ts honors stopReason (truncation → raise maxTokens or fail with the real reason)
  • L5 plan schema maxItems + bounded default governance (plan-schema.ts, router.ts)
  • L6 formattingDensity must not count plain newlines (reward-model.ts:82-88)
  • L7 review-queue pruning/compaction + snapshot size bounds (review-queue.ts, plane-persistence.ts)
  • L8 aggregateFeedback homogeneity check (feedback.ts:50-64)
  • L9 rubric-version pinning into GateVerdict, editorial history, and gold entries (also R21 prerequisite)
  • L10 per-content-type editorial checklists bound and consumed (EditorialChecklistEntry)

Phase 3 — Theme 1: media quality plane#

  • R1 VLM-as-judge for images feeding qualityAggregate + domain-routed criticFor (OpenRouter multimodal; fail-loud without creds)
  • R2 cross-modal consistency judge (VLM caption/VQA + embedding cosine vs brief); media-aware compellingness judge
  • R3 audio probes: ffmpeg ebur128/silencedetect/ffprobe duration on narration + music; independent measuredLufs (ASR WER: seam)
  • R4 video probes: ffprobe duration/resolution, frame sampling → existing image moderation, SSIM flicker variance
  • R5 media best-of-N with seed control (K jobs, VLM selection, losers cataloged unreleased; seed threaded to providers)
  • R6 structured media prompts: per-stylePreset style-guide table + optional LLM prompt-director pass
  • R7 provider quality routing: per-kind registry, measured win-rate/latency/cost, weighted routing + fallback (stats sink now)
  • R8 caption/dub faithfulness seam (ASR → back-translation similarity) (seam)

Phase 4 — Theme 2: learning-loop back half#

  • R9 preference-pair persistence (tournament + best-of-N + human verdicts → JSONL export, DPO/RM-ready)
  • R10 gold-set → judge-calibration job (assemble, holdout, re-score, κ, recalibrate bars)
  • R11 human post-edit diff channel into the preference learner
  • R12 per-dimension human verdicts (queue API + per-dimension calibration)
  • R13 Platt/isotonic calibration (= M3)
  • R14 engagement-outcome attribution poller binding (seam over a metrics source interface)

Phase 5 — Theme 3: judge robustness#

  • R15 heterogeneous panels (= H2) + κ-weighted aggregation option
  • R16 injection fencing + red-team suite (= H3)
  • R17 production judging samples ≥ 2 + fixed-operating-point variance (= L1) in the BFF plane wiring
  • R18 cost-aware cascaded judging (cheap screen → full panel near threshold/variance)
  • R19 exemplar-anchored judging (scored gold exemplars in rubric prompts); optional retrieval-augmented judging seam
  • R20 reconciliation/debate round before human escalation
  • R21 rubric-version regression gating (champion-challenger over judge versions vs gold benchmark) + version pinning (= L9)

Phase 6 — Theme 4: production-time search#

  • R22 plan-quality judge before execution (reject/re-plan below bar)
  • R23 multi-plan sampling + argmax selection
  • R24 node-level best-of-N + best-draft tracking (= M11)
  • R25 dimension-targeted critics (per-dimension scores + evidence + compare-to-prior) + full critique trajectory threading
  • R26 retrieval tools for writer nodes (AgentLoop-backed generator over Sophia BM25; verifiable citations)
  • R27 replanning on node failure (repairPlan, requireMetDependencies policy)
  • R28 parallel frontier execution + context budgeting (summarizeDependencies) + content-hash cache + seedArtifacts resume
  • R29 per-node model routing (modelFor(node))

Phase 7 — Theme 5: ideation & portfolio#

  • R30 multi-candidate ideation (N synthesizer samples, dedupe, union)
  • R31 semantic embedding novelty seam (injectable TopicEmbedder; CPU MiniLM binding where runnable)
  • R32 OPRO-style rejection-reason mining → rubric/brief-template steering suggestions
  • R33 cross-cycle per-category style memos (versioned, injected into briefs)
  • R34 persona/audience simulation judges per category audience card
  • R35 uncertainty-aware compute routing (escalate low-confidence, per-category stakes scaling)
  • R36 exploration floor (min-trial quota) + contextual bandit (LinTS behind CategorySampler, Beta cold-start fallback)
  • R37 wire planCalendar + week-level angle-mix variety objective (= M21 plane side)

Phase 8 — Theme 6: editorial & post-publish#

  • R38 bind publish-readiness pipeline + per-type checklists (= H5, L10)
  • R39 A/B holdout publishing machinery (segment selector, paired McNemar readiness; live traffic = deploy-bound) (seam)
  • R40 staleness re-review sweeps over published artifacts (recurrence reuse)
  • R41 inter-rater reliability: sampled dual-assignment + per-pair κ scorecards (real κ math now; dual reviews accrue at deploy)
  • R42 post-publish quality monitoring seam (corrections/complaints/ engagement → ReviewOutcome + auto re-review entry)