Disciplines · Audits

V1 Autonomous Content Systems — SOTA Audit — 2026-07-02

discounted Thompson sampling with exact Beta-preserving decay, Bradley–Terry tournaments with double-ordering position-bias mitigation, pessimistic-LCB best-of-N, corpus-level diversity gates, coverage-discounted panel confidence,

5sections19 minread
Markdown source
On this page

Scope: every system that lets V1 autonomously produce content — the creative-autonomy director plane, the creative orchestrator + @oshun/ai agent-loop, the BFF generation executors/Isis release gate/autonomy bindings, the shared content-quality-judge engine + release gates + content-service, and the editorial/governance layer (studio-authoring, generation-control-isis, agentic-studio, agent-pipelines).

Method: five deep-read adversarial audit agents (max 2 concurrent, per session limits), each reading every core source file with the repo's stub-indicator battery plus an explicit 2026-SOTA technique checklist; every HIGH finding was then independently re-verified by the coordinating session against the source before being recorded here. Prior ledgers (CREATIVE_AUTONOMY_UNDERLYING_AUDIT_2026-07-02.md, AGENTIC_SYSTEMS_FULL_AUDIT_2026-07-02.md, V1_V9_AGENTIC_CONTENT_SOTA_ASSESSMENT_2026-06-14.md) were used as the baseline so this pass verifies rather than re-discovers.

Verdict up front#

The text path is real and in several places ahead of common 2026 practice — discounted Thompson sampling with exact Beta-preserving decay, Bradley–Terry tournaments with double-ordering position-bias mitigation, pessimistic-LCB best-of-N, corpus-level diversity gates, coverage-discounted panel confidence, fail-closed governance with content-hash-bound promotion. The adversarial stub scan across all five units returned zero result-faking hits; tests assert computed values throughout.

But five verified defects undermine the quality loop as deployed, and the single biggest SOTA gap is structural: the quality engine is text-only. V1 generates images, narration, music, and video, yet no automated system ever looks at the pixels or listens to the audio — qualityAggregate is hardcoded null in all six provider adapters, the only "judge" media meets is a text LLM scoring the job's JSON metadata, and the plane's revise-with-feedback loop is therefore blind for media. Meanwhile the learning loop's back half is dormant: tournament preference pairs are discarded, promoted gold sets are never consumed for judge calibration, and human post-edit deltas — the strongest quality signal available — are captured but never learned from.

§1 Verified defect ledger#

Severity ordering. Every HIGH was re-verified line-by-line by the coordinator; MED/LOW items carry the deep-read agent's evidence and were spot-checked where load-bearing.

HIGH#

  • H1. Restart double-learning corrupts the bandit/calibration state. libs/oshun/creative-autonomy/src/director/director.ts:218 keeps learnedItemIds as a private in-memory Set; apps/oshun/bff/src/agentic/autonomy-bindings/plane-persistence.ts:42-57 snapshots/restores the queue, sampler arms, and learner — but not that set. After every restart, ingestResolutions() (director.ts:683-713) re-feeds the entire resolved history into posteriors and calibration EMAs that already contain those updates. Posteriors artificially concentrate (killing exploration) a little more on each deploy. (verified)
  • H2. The production judge panel is three copies of the same model. apps/oshun/content-service/src/main.ts:33-43 builds all three JudgePanel members from one provider and one judgeModel, varied only by temperature (0.2/0.4/0.6). Median-of-correlated-draws ≈ a single judge, so the disagreement→needsHuman escalation, LCB pessimism, and active-learning routing all systematically underestimate uncertainty. The lib's own header (judge-panel.ts:4-6) prescribes "different model and/or rubric framing"; distinct model chains are already pinned in model-routing.ts. (verified)
  • H3. No prompt-injection fencing on judged text, and no judge red-team suite. libs/shared/content-quality-judge/src/judge-engine.ts:207-213 (and the pairwise path, and player-proxy.ts) embed the artifact after a bare TEXT: with no delimiters and no "never follow instructions in the text" clause; a grep for fencing language across the lib returns nothing. In a self-gating autonomy pipeline, a candidate that captures the judge gates its own release — and best-of-N then preferentially selects injected candidates. (verified)
  • H4. Job-drain double-execution race. apps/oshun/bff/src/generation/jobs-route.ts:142-159 snapshots status === 'queued' once and never re-checks before flipping each job to running; the loop awaits between jobs. Concurrent drains — operator tick, autonomy producer runJob, cycle driver are all callers — can both capture the same job: double provider spend, result/governance clobber, duplicate gallery records (output-catalog.ts:182), and the corollary runJob failure "unknown error" on reading a running job. (verified)
  • H5. The publish-readiness pipeline has zero production callers. libs/oshun/studio-authoring/src/editorial-lifecycle/pipeline.ts:50,155 (evaluatePublishReadiness — Sophia evidence pack, Lilith tone-review id, Isis release-gate ids, rights/provenance bundle, localization ratio — and buildPublishingPipelineDecision) are defined and tested but unbound; the autonomous publisher's default gate set is a single Lilith tone gate (apps/oshun/bff/src/agentic/autonomy-bindings/editorial-publisher.ts:81). An autonomous text item can publish with no evidence pack, no rights bundle, no localization check. (verified — grep shows definitions only)

MEDIUM#

Creative-autonomy plane

  • M1. Unbounded repeat feedback: autonomy-route.ts:545-597 + learner.ts:177-186 have no per-(user,item) dedupe or rate cap — one user can pump a category's posterior arbitrarily (found independently by two agents).
  • M2. Failed-topic cooldown keyed to submittedAtUnixSeconds, not resolution time (director.ts:375) — an item rejected 6.9 days after submission gets ~0 effective cooldown.
  • M3. Calibration leniency ratchet: with no explicit rating, humanEquivalent = the outcome reward constant (learner.ts:155-159); approving a mediocre 0.55-score item pushes the category's additive offset up by λ·0.45 — a lenient reviewer progressively opens the gate for everything in the category.
  • M4. Gate-retry can discard a judged champion: if the retry attempt yields zero complete variants, director.ts:558-575 reports production-failed and the attempt-0 below-bar champion never parks for human review. (LOW-MED) Related accounting: completeVariantCount holds only the last attempt while variantCount accumulates all attempts (director.ts:556-557).
  • M5. LLM-synthesized topics bypass sanitizeTopic (ideation.ts:241-250 admits proposal.topic verbatim; source-signal topics get the 160-char single-line defense at signals.ts:100-101). (LOW-MED)

Judge engine / content-service

  • M6. Smart-apostrophe blindness in the slop detector: slop.ts:64,84,90-91 patterns are byte-exact straight-apostrophe ("couldn't help but"), but LLM prose emits U+2019 — several high-weight tells silently never fire.
  • M7. Run-id collision overwrites audit records: content-pipeline-service.ts:95-97 derives run_${briefId}_1 deterministically; resubmission overwrites the prior persisted run.
  • M8. No compute ceiling on the §3.2 HTTP path: http-router.ts:32-40 validates 4 strings; candidateCount has no upper bound and the compute-budget meter is never wired into the dispatcher — one POST can trigger thousands of writer+panel calls.
  • M9. The deployed content-service mount omits the grounding gate entirely (main.ts:81-86) — canon-inventing prose ships ungated in the one production wiring.
  • M10. external-benchmark.ts:152-216 labels authored fixtures with real dataset identities (eq-bench-creative-v3, litbench) and invented publishedScores — honest header, dishonest-at-a-distance reports.

Orchestrator / agent-loop

  • M11. Last-draft-wins revision: critique-revise.ts:66-83 returns the final iteration unconditionally — a revision scoring 0.4 ships over a 0.75 draft 0. Same pattern in libs/shared/ai reflexion.ts:103-115. Cheap fix (track argmax), high content-quality impact. (verified) Note: the shared judge lib's self-refine.ts:107-116 does this correctly; the orchestrator the plane actually uses does not.
  • M12. Self-grading deterministic critic: default-critic.ts:74-185 reads measuredLufs, goldSourceIds, referenceImage from the artifact being graded — a generator that echoes the gold set passes trivially. Grading references belong on node.params or an independent measurement seam.
  • M13. Governance spend accounting is post-hoc and success-only: router.ts:108-117 records a dispatch only on success and at a flat per-domain estimate; revision iterations and judge tokens are invisible to maxDispatches/maxCostUsd. Also options.signal is never checked in the dispatch loop (router.ts:60-126) — aborting a run keeps dispatching.
  • M14. Residual banned-pattern stubs in libs/shared/ai/src/prompts/ testing.ts:732,794-812,933-937 (placeholder quality eval, pValue = isSignificant ? 0.01 : 0.5) and advanced/index.ts:456-460 ("In production, would use embeddings"). Off the orchestrator path but exactly the pattern the repo bans — should be gutted to fail-loud.

BFF generation / autonomy bindings

  • M15. The media autonomy path is deterministically dead while advertised live: server.ts:1158-1164 marks image/audio/video plannable on credential presence, but every adapter ships measurements that guarantee the Isis gate blocks or holds (provenance false, watermark null vs floors 0.99–1.0, qualityAggregate: null in all six adapters — verified), and runJob throws on both blocked and needs_review (generation-producer.ts:52-57). With real keys the plane pays providers every media cycle and discards every artifact. Honest fail-closed governance — but an economic drain plus a misleading "live domains" signal until the deploy-bound watermark/signing steps exist.
  • M16. needs_review conflated with failure in the autonomy producer — a gate verdict of "held for a human" aborts the variant instead of routing the held media into the plane's review queue.
  • M17. Media "judged" by JSON metadata: create-plane.ts:125 gives all domains createLlmJudgeCritic, which serializes {jobId, kind, url, provider} into a text prompt — a real LLM call producing an epistemically meaningless score, which then drives the revise loop.
  • M18. Latent analysis gate bypass: release-gate.ts:132-138 releases releaseKind:'analysis' outputs with no url ungated, but narration already returns inline audioBase64 with no url — any future executor tagging inline media analysis skips the gate silently. (LOW-MED)
  • M19. Unvalidated executor casts (request as IllustrationCardInputs, image-executor.ts:61; admin raw-enqueue accepts any body, jobs-route.ts:288-296) → "undefined — undefined style" prompts sent to paid providers. (LOW-MED) Provider misattribution: image-executor.ts:67 stamps provider: 'runpod' for the Stability client. (LOW)

Editorial / governance

  • M20. Gold-set consumption dormant: goldSetFromEntryRecords, heldOutSplit, queryEntries have no production consumers — operators can promote entries but assembled sets never reach judge calibration (MED-HIGH: the learning loop's back half doesn't run).
  • M21. Editorial calendar machinery dormant: evaluateEditorialPublishWindow, embargo, recurrence (lifecycle.ts:176-208) have no callers; the director publishes immediately on approval (director.ts:271-273). The plane's own planCalendar (selection.ts:193-228) is likewise real, tested, and never called.
  • M22. Voice-disclosure gate missing: release-gate-model.ts:20-23 promises spoken-synthesis-disclosure-audible / voice-watermark-every-turn / watermark-every-frame; only generic watermark-coverage floors exist and no audible-disclosure gate at all.
  • M23. Champion-challenger uses an independent two-proportion z-test on paired samples (champion-challenger.ts:59-66) — McNemar is the correct paired test; p-values are miscalibrated. (LOW-MED) Editorial records/transition history are in-process Maps — restart erases the audit trail of which gates each published item passed (editorial-publisher.ts:83,110). (LOW-MED)

LOW (selected)#

  • Pointwise "variance" conflates the temperature ladder (t, t+0.1, t+0.2) with judge noise (judge-engine.ts:220-224).
  • cohenKappa/pearson silently truncate mismatched arrays (calibration.ts:66-67,86).
  • Welch drift on n=1 windows → z=±Infinity → spurious quarantine (drift.ts:72-81); delta===0 counted as 'better' in editorial drift (drift-detection.ts:60-65).
  • structured-output.ts:119-154 ignores stopReason — truncated JSON burns all retries at the same maxTokens.
  • Plan schema has no maxItems on nodes; default governance gate is ALLOW_ALL_GATE (plan-schema.ts:21-24, router.ts:54).
  • formattingDensity counts plain newlines as formatting markers (reward-model.ts:82-88).
  • Review queue grows unboundedly and the full store (variants included) is serialized on every persist (review-queue.ts:114-183, plane-persistence.ts:42-50).
  • aggregateFeedback blends mixed-bundle ratings silently (feedback.ts:50-64); @oshun/agent-pipelines is a 2-line re-export facade; EditorialChecklistEntry has zero consumers.

§2 SOTA gaps — where further techniques/layers/algorithms would raise quality#

Consolidated across the five units, deduplicated, ranked by expected content-quality impact. "Now" = implementable on this box today; "gated" = needs live creds/data/GPU.

Theme 1 — A media quality plane (the largest gap: nothing looks at the pixels)#

  1. VLM-as-judge for images — score composition/artifacts/text-legibility/ prompt-fidelity by sending the produced URL through the existing OpenRouter multimodal path. Slots: compute inside adaptImageProviderGenerate (image-provider-env.ts) feeding the currently-null qualityAggregate, plus a domain-routed criticFor(node.domain) map replacing the blind JSON critic in create-plane.ts:125. M, now (same key that gates the plane).
  2. Cross-modal consistency check — VLM caption/VQA of the produced image → embedding cosine vs the brief + dependency text; a new media-aware CompellingnessJudge (all five current judges throw on media-only variants — variant-text.ts:16-52). M, now.
  3. Audio quality probes — ffmpeg is installed on-box: ebur128 loudness, silencedetect, ffprobe duration-vs-request on narration (audioBase64) and music URLs; emit measuredLufs/duration so the orchestrator's existing deterministic critic consumes an independent measurement (also fixes M12 for audio). ASR round-trip WER: gated. Rest: M, now.
  4. Video probes — ffprobe duration/resolution vs request; sample N frames → per-frame moderation with the existing image safety classifier (closes video's safetyScanScore: null); frame-pair SSIM variance for flicker. Slot: adaptVideoProviderGenerate. M-L, mostly now.
  5. Best-of-N for media with seed control — enqueue K=2-4 jobs with varied seeds in the image/video DomainGenerator, select by the VLM scorer; record losers as non-released candidates. Add seed to provider requests for reproducibility (nothing threads one today). M, now; spend-gated in degree.
  6. Structured media prompt construction — replace "${subject} — ${stylePreset} style" (image-executor.ts:43) with a declared per-stylePreset style-guide table (palette, composition, negative prompts — deterministic brand consistency) plus an optional LLM "prompt director" expansion pass in the autonomy media generators. S-M, now.
  7. Provider quality routing — provider-measurement.ts builds envelopes but measures nothing and keeps no history; one hardcoded provider per kind, no fallback. Per-kind registry with measured win-rate/latency/cost feeding weighted routing. L, partially data-gated.
  8. Caption/dub faithfulness — ASR the dubbed track → back-translation → semantic similarity vs source. L, gated (ASR creds).

Theme 2 — Close the learning loop's back half#

  1. Preference-pair persistence (DPO/RM-ready) — tournament ComparisonRecords and losing variants are computed then discarded (director.ts:577-579), and best-of-N ranked pairs likewise; persist {brief, chosen, rejected, margin, source: tournament|human, provenance} JSONL. This is free training data already being paid for. Slots: after runVariantTournament + a preference-export.ts in the judge lib. S-M, now (training itself GPU/creds-gated).
  2. Wire the gold-set → judge-calibration loop (M20) — scheduled job: assemble per-family sets, heldOutSplit, re-score holdout with the panel, compute κ via the existing calibration.ts, recalibrate per-category gate bars. All machinery exists; nothing runs it. M, now.
  3. Human post-edit diff channel — editedContentRef and computeOutputDiff exist but the learner has no edit channel; on edit-kind decisions compute changed-fraction + section diffs, feed as shaped reward and store as prompt-revision exemplars. The strongest quality signal there is. S-M, now.
  4. Per-dimension human verdicts — judges emit 7 dimensions; humans return one scalar (autonomy-route.ts:265-267). Add per-dimension agree/override chips → per-dimension calibration instead of one additive EMA. S, now.
  5. Replace the additive calibration offset with Platt/isotonic scaling per category once observations ≥ N (fixes M3's leniency ratchet structurally); keep the EMA as cold-start. Slot: learner.ts:152-190. M, now.
  6. Engagement-outcome attribution — published-item live metrics → fractional bandit reward + calibration observations; full-auto categories currently learn nothing at all (director.ts:695). Slot: a poller binding beside createAutonomyCycleDriver. M, data-gated.

Theme 3 — Judge robustness#

  1. Heterogeneous judge panels + agreement weighting — route the three panel members through distinct pinned model chains (fixes H2; routes already exist in model-routing.ts); later add per-member κ-derived weights (inverse-variance pooling). S now; weights data-gated.
  2. Injection fencing + a judge red-team regression suite — delimit judged text, add instruction-immunity clauses, and ship a judge-red-team.ts probe library (score-inflation injections, rubric echo, flattery, homoglyph slop evasion, markdown stuffing) run as a CI gate with scripted providers (fixes H3). S-M, now.
  3. Turn on multi-sample judging in production — the discipline exists (samples option, temperature-perturbed means) but the BFF wires the default samples: 1 (create-plane.ts:130). Also fix the temperature ladder so variance is measured at a fixed operating point. S, now.
  4. Cost-aware cascaded judging — cheap screen (single fast judge + slop/lexical heuristics) auto-resolves scores far from threshold; full panel only near the bar or on high variance; pairs with the active-learning band. M, now.
  5. Exemplar-anchored judging — append 2-3 scored gold exemplars per content type (already curated in benchmark.ts) to renderRubric; optional retrieval-augmented judging (canon passages in judge context). S, now.
  6. Reconciliation/debate round before human escalation — on disagreement > threshold, each judge sees the others' scores+reasons and re-scores once; escalate only if still split. Directly lowers human-touch rate. M, now.
  7. Rubric-version regression gating — rubric semver exists but nothing forces a κ-non-regression run on the gold benchmark when a rubric/prompt/slop-list bumps; reuse champion-challenger with the judge as the variant. Pin rubric versions into GateVerdict, editorial history, and gold entries (one field each). M, now.

Theme 4 — Production-time search (plan → draft → revise)#

  1. Plan-quality judging before spend — plans are only structurally validated (planner.ts:97-104); an LLM plan-critic (coverage vs brief, node atomicity, dependency sanity) rejecting/re-planning below bar is the cheapest quality lever in the loop. S-M, now.
  2. Multi-plan sampling — sample 2-3 decompositions at temperature, score with the plan critic, pick argmax (ToT-lite). S-M, now.
  3. Best-draft tracking + node-level best-of-N — fix M11 (argmax draft), then K parallel first-pass samples with the critic picking the revision seed. Slot: critique-revise.ts. S, now.
  4. Dimension-targeted critics — JUDGE_SCHEMA is {score, feedback}; extend to per-dimension scores + evidence quotes + compare-to-prior so revision feedback is targeted, and thread the full critique trajectory (feedbackHistory) instead of last-critique-only. S, now.
  5. Retrieval tools for writer nodes — the fully-built AgentLoop/ToolRegistry is never used for generation; an AgentLoop-backed text generator with retrieval (Sophia BM25 exists in-repo) lets writers ground and cite honestly — and fixes half of M12 by making citations verifiable. M, now.
  6. Replanning on node failure — failed nodes cascade-fail dependents with no retry/repair; add repairPlan(plan, failures) re-decomposition and requireMetDependencies policy. M, now.
  7. Parallel frontier execution + context budgeting + cache/resume — Kahn-frontier scheduling with maxParallelNodes; budgeted summarizeDependencies per node token allowance (yemaya adapter JSON-dumps full upstream outputs today); content-hash cache on the generator registry and routePlan(plan, {seedArtifacts}) resume. M each, now.
  8. Per-node model routing — strong model for prose nodes, cheap for outline/metadata; modelFor(node) in the orchestrator config. S, now.

Theme 5 — Ideation & portfolio intelligence#

  1. Multi-candidate ideation — one synthesizer call per cycle today; N independent samples at varied temperature, deduped by topicSimilarity, union into candidates. S, now.
  2. Semantic embedding novelty — acknowledged seam (ideation.ts:318-320); trigram Jaccard misses paraphrase duplicates ("blood moon" vs "total lunar eclipse"). Injectable TopicEmbedder (MiniLM-class CPU embeddings run on this box) for novelty scoring, cooldown matching, and cluster guarding. S, now.
  3. OPRO-style rubric/brief evolution — mandatory rejection reasons and tweak directions are stored but never mined; a periodic reflection job clusters them per category into proposed rubric addenda and brief-template mutations, emitted as SteeringSuggestions (human-owned, tighten-only preserved). M, now.
  4. Cross-cycle style memory — per-category StyleMemo (LLM-summarized contrast of approved vs rejected champions, versioned like steering) injected into composeBrief; tweaks are currently one-shot and champion traits are never distilled. M, now.
  5. Persona/audience simulation judges — all judges score generic quality against one hardcoded register ("curious general audience"); persona-conditioned judges ("would this reader click/finish/share?") per category audience card slot directly into the partial-panel machinery. S, now.
  6. Uncertainty-aware compute routing — confidence currently only fails the gate; escalate confidence-only failures to more samples/judges, route persistent low confidence to humans rather than parking, and add per-category stakes scaling variants/samples (value-of-compute). S-M, now.
  7. Contextual bandits + exploration floor — LinTS/logistic TS over signal-kind/timeliness/novelty features behind the existing CategorySampler interface once volume justifies it (M-L, data-hungry); meanwhile a min-trial quota so low-prior categories are never starved (S, now).
  8. Wire calendar-level portfolio planning — planCalendar is built, tested, and never called (M21); schedule approved publications and add a week-level angle-mix/variety objective. S wire / M objective, now.

Theme 6 — Editorial & post-publish#

  1. Bind the publish-readiness pipeline (H5) so autonomous publishes require the evidence pack/rights/localization gates by content type; bind per-content-type EditorialChecklistEntry data. S-M, now.
  2. A/B or holdout publishing — publish 2 gate-passing variants to segments, measure, feed back (reuse champion-challenger with the McNemar fix). M-L, data-gated.
  3. Staleness re-review sweeps — reuse the dormant recurrence machinery to create re-review calendar entries for aging published artifacts. S-M, now.
  4. Inter-rater reliability of humans — sampled dual-assignment in the review queue + Cohen's κ per reviewer pair (no human-vs-human agreement is measured anywhere). M, data-gated.
  5. Post-publish quality monitoring — corrections, complaints, engagement regressions → ReviewOutcome + auto re-review. M-L, data-gated.

§3 What is already at or beyond SOTA (keep and build on)#

  • Discounted Thompson sampling with Beta-preserving decay + two-level (category, angle) arms + exact Marsaglia–Tsang sampling (bandit.ts:136-144, selection.ts:91-107).
  • Bradley–Terry MM tournament + both-orderings position-bias mitigation with inconsistency→tie collapse (tournament.ts:58-96, llm-judges.ts:179-186, judge-engine.ts:292-326).
  • Pessimistic-LCB best-of-N resisting reward-hacking, plus excess-correlation reward-hacking detection that auto-hardens selection (best-of-n.ts:60-65, reward-model.ts:173-190).
  • Corpus-level diversity gates failing homogeneous batches whose items individually pass, plus generation-time novelty pressure (corpus-gate.ts, diversity-pressure.ts).
  • Partial specialist panels with coverage-discounted confidence and typed refusal of uncovered dimensions (rubric.ts:147-164).
  • Fail-closed single-path governance: one executor→Isis-gate→catalog drain for operator, curated, and autonomous jobs; evidence-mandatory release gates with content-hash-bound promotion and named signoff; human-review trigger catalog where an unassigned reviewer blocks release.
  • Honest fail-loud posture everywhere the audit looked: zero result-faking stubs on the production paths (the only exceptions found are the M14 files, which are unreachable from these paths).
  • AIMD concurrency governor honoring provider retry-after; durable snapshots for jobs/catalog/plane; full provenance chains (signal → idea → scores → tournament → gate) on every review item.
  • Statistically real drift (direction-aware Welch z), κ-calibration with textbook-value tests, champion-challenger promotion gates, salted-SHA-256 deterministic holdout splits.

§4 Priority worklist#

P0 — verified defects (fix before further feature work)

  1. H4 drain race (re-check status === 'queued' per job; or a claim flag).
  2. H1 restart double-learning (persist learnedItemIds, or derive from learner state).
  3. H2 heterogeneous panel (route members through the pinned distinct chains).
  4. H3 fencing + minimal red-team probes.
  5. H5 bind publish-readiness gates for autonomous publishes.
  6. M11 best-draft argmax; M16 route needs_review media to the review queue; M6 apostrophe normalization; M9 grounding gate in the deployed mount; M1 feedback dedupe; M14 gut the residual prompts/ stubs to fail-loud.

P1 — highest quality-per-effort, implementable now Media plane first (items 1-6): VLM image judge + cross-modal check + ffmpeg audio/video probes + media best-of-N + style-guide prompts — this converts media from "safety-gated only, judged blind" to first-class judged content. Then learning-loop back half (9-13): preference-pair logging, gold-set→κ job, per-dimension verdicts, post-edit diffs. Then judge robustness (15-21) and production search (22-29): plan judging, multi-plan, retrieval-grounded writers. Then ideation/portfolio (30-35, 37).

P2 — data/creds/GPU-gated (build seams now, bind at deploy) Engagement attribution (14), contextual bandits (36), DPO/RM training on the logged pairs, ASR-based WER/dub checks (8), A/B publishing (39), inter-rater κ (41), post-publish monitoring (42), live κ calibration on real provider traffic.

Method notes#

  • Five audit units: libs/oshun/creative-autonomy; libs/oshun/creative-orchestrator + libs/shared/ai + libs/shared/content-eval; apps/oshun/bff/src/generation + src/agentic; libs/shared/content-quality-judge + libs/shared/content-release-gates + libs/oshun/content-service; libs/oshun/studio-authoring + libs/oshun/generation-control-isis + libs/oshun/agentic-studio + libs/oshun/agent-pipelines.
  • Each unit: full read of core sources, stub-indicator grep battery, verification of prior-ledger claims (49/49 prior claims checked: 45 CONFIRMED, 4 PARTIAL — the PARTIALs are recorded above as M-items or caveats; 1 claim REFUTED at the orchestrator layer: "revision kept only on improvement" = M11).
  • All five HIGHs re-verified line-by-line by the coordinating session before inclusion. MED/LOW findings are single-deep-read unless marked verified.
  • No code was modified in this pass — audit only.