Latency, Accessibility, Eval Sets, and Tests

This page documents the quality spine of Living Scenes: the latency and backpressure budgets that keep a live scene responsive, the backend resolver that chooses between the chained-clip pipeline and the feature-flagged Phase 176 world-model, the reduced-motion and accessibility companions every Score ships with, and — most concretely — the per-template eval sets, release gates, and the deterministic test contracts that decide whether a workflow class is allowed to promote to the contemplative tier at all. It serves the AAA-Creator and operator authoring consoles (who watch these gates before they ship), trust-and-safety review (whose crisis and abuse signals feed the eval corpus), and the contemplative product (whose users feel the latency and accessibility budgets directly). In the V1 stack it sits downstream of the Yemaya generation substrate and the governance spine documented in Scene Safety, Determinism, Provenance, and Cue Privacy: where governance decides whether a frame is safe, this page decides whether a whole workflow class renders well enough, fast enough, accessibly enough, and deterministically enough to ship. Much of it is real, tested decision logic in libs/yemaya/living-scenes-runtime/src/ (the published package @yemaya/living-scenes-runtime) and libs/oshun/trust-safety/src/; the latency numbers, the continuity-metric models (FVD, optical-flow morph), and the real pixel-level renders are upstream of these gates, and this page says so plainly wherever it matters.

Latency, Quality, and Backpressure Budgets#

A Living Scene is a continuous AI-generated video that listens — the user shapes it live by voice, tap, or text — so the latency budget is the difference between a contemplative experience and a stuttering one. The V1 targets are:

• Blend-window quality: each blend window carries a per-frame temporal-flicker score and a Fréchet Video Distance (FVD) across the window. When either crosses the workflow class's threshold, the runtime re-renders the boundary or falls back to a longer blend window rather than shipping a visible seam.
• Backpressure is a graceful-degradation ladder, not a failure: when GPU latency degrades, the runtime first extends the current segment, then reduces the next segment's resolution within tolerance, then lengthens the blend window, and only then falls back to still-image-plus-narration — it engages these before failing the session. The terminal rungs of this ladder are the same fallbacks the Psyche envelope surfaces as scene.fallback-engaged events (see below).
• Quality dashboards track P50/P95/P99 per region, per provider, per template, and per persona, fire SLO-breach alerts, and map a breach to the corpus of active sessions it impacts so an operator can see who is affected, not just that a number moved.

Honest status: the latency targets and the backpressure ladder are product-spec contracts. The numbers (P50 ≤ 400 ms, FVD thresholds, optical-flow morph quality) are met by upstream rendering infrastructure that is not part of the decision-logic libraries this page is grounded in; the governance and eval code below is the genuinely-implemented, tested part of the story.

Backend Strategy and the Phase 176 Bridge#

The backend that actually produces frames is chosen by resolveBackend in libs/yemaya/living-scenes-runtime/src/backend/backend-resolver.ts. There are exactly two backends (BACKEND_IDS):

• chained-clip — the V1 default and the only backend on the contemplative tier at launch. It is the continuity-disciplined pipeline: parallel pre-warm of N-second segments, latent warm-start, optical-flow morph, LUT match, and audio crossfade across overlap windows. planChainedClip constructs its ChainedClipPlan (preWarmSegments, blendKernelVersion, mode: 'materialize-then-stream' | 'stream-live') and requires minLookaheadSegments ≥ 2 so there is always a segment warming behind the playhead.
• phase-176-nous — a feature-flagged alternate that swaps the chained backend for a live latent world-model from the Nous domain's Phase 176 serving program. The customer surface is unchanged; only the substrate differs.

The selector is keyed on a four-value entitlement tier (ENTITLEMENT_TIERS):

For non-contemplative tiers, the resolver sorts the workflow class's supportedBackends by BACKEND_QUALITY_RANK (chained-clip: 0, phase-176-nous: 1) and picks the highest-rank backend that both the phase176FeatureFlag enables (Phase 176 is skipped when the flag is off) and the per-backend SLO canServeNow confirms it can serve under headroom. The contemplative branch is a separate, explicit code path: it throws contemplative-cannot-resolve-phase-176 / unknown-backend rather than ever letting a contemplative session touch the world-model backend. When nothing can serve, it throws no-backend-can-serve instead of silently degrading. This is exactly the Backend swap test contract below: AAA-tier sessions can flip between the two backends without breaking the customer surface, while contemplative-tier sessions can never resolve to Phase 176.

Reduced-Motion and Accessibility Renders#

Accessibility is not a post-hoc variant; every Score produces two renditions in lockstep. The runtime enumerates them as LIVING_SCENE_RENDITIONS in libs/yemaya/living-scenes-runtime/src/personal-artifacts/personal-artifacts.ts:

• full-motion — the continuous video.
• reduced-motion — a still-image carousel that advances at the same per-segment cadence, with the same narration and ambient audio. It is modeled as ReducedMotionSceneSegment records (segmentId, durationSeconds, stillFrameUrl, narrationAudioCue, ambientAudioCue). Reduced-motion is an independent shareable artifact and a vestibular-friendly accessibility companion — it is not a degraded second-class copy.

On top of the two renditions:

• Captions: a sidecar caption track per locale, with optional burn-in for channels that strip sidecars (default off, user-toggle).
• Audio description (AD): a per-segment AD track narrating the visual content for screen-reader users; it ships with every template.
• Transcript: a full transcript of narration + AD, linked from the public viewer and downloadable.
• Color-contrast and font-size: the public viewer respects OS-level accessibility settings and offers high-contrast and large-text chrome variants.
• Keyboard and assistive-tech navigation: cue verbs are reachable by keyboard, and the scrubber, share, and accessibility controls are usable without a pointer.
• Accessibility release gate: promoting any workflow class to the contemplative tier requires the reduced-motion variant + AD + captions + transcript to pass parity QA. This is one of the gates enumerated below.

The fallback ladder and accessibility renditions are observable on the Psyche envelope: SceneFallbackEngaged in libs/oshun/embodiment-psyche/src/events/scene-events.ts carries a fallbackKind of reduced-motion-render | still-image-only | transcript-only | audio-with-still | pre-rendered-substitute and a trigger of provider-degraded | policy-rejection | eval-gate-fail | shareability-restriction. A reduced-motion fall-down under provider degradation and a reduced-motion fall-down under an eval-gate-fail are the same render path surfaced with different triggers, which is why the accessibility artifact has to be a first-class, always-present rendition rather than something synthesized on demand.

Eval Sets, Release Gates, and Rollback#

A workflow class earns the contemplative tier by passing a battery of evals against a per-template fixture corpus. The corpus is ≥ 30 fixtures per template at V1, spanning six fixture families: golden-path, adversarial cue, crisis-trigger, locale-parity, accessibility-parity, and grounding-correctness (the last for Nyx / Veritas / Metis templates — see Veritas — Grounded Stories and Claims, Nyx — Sky Events and Perspective, and Metis — Education and Tutoring).

The eval families and the code each gate runs#

Release gate: every workflow class must pass all six — continuity, determinism, watermark, cue policy, crisis, and accessibility — before promotion to the contemplative tier. Failures block promotion and are surfaced in the Isis release-gate dashboard (see Isis Generation Control). This is the backstop §25 contract; the backlog framing is §25 in ../TODOS.md.

Rollback: any production workflow class can be rolled back to a prior version. In-flight sessions complete on the current version; new sessions resolve to the rolled-back version; and kept artifacts pinned to a version remain playable under their pinned render envelope. This is the Determinism test's third leg — a rollback must leave pinned artifacts rendering identically.

Determinism harness — the per-frame SHA-256 gate#

checkDeterminism in libs/yemaya/living-scenes-runtime/src/determinism/determinism-harness.ts is the actual decision function behind the determinism gate. It is a pure function over per-(node, frame) SHA-256 digests — the rendering happens upstream; the harness only judges the hashes. Each NodeFrameDigest is { nodeId, frameNumber, sha256 }; the harness groups digests by frame, and any frame where the set of node hashes has more than one distinct value is a mismatch. It returns a DeterminismReport:

interface DeterminismReport {
  nodeIds: readonly string[];
  frameCount: number;
  mismatchedFrames: readonly number[];
  perNodeMismatchCount: ReadonlyMap<string, number>;
  pass: boolean;
}

Two edge cases are load-bearing. First, a run with fewer than 2 nodes auto-passes (pass: true) — there is nothing to disagree with, so a single-node render can never fail determinism, only be inconclusive. Second, the gate is pass iff every node's mismatch count stays within mismatchToleranceFrames; the per-node count is measured against the first sorted node as the reference hash. A single node that drifts beyond tolerance fails the whole gate. This is intentionally a tolerance gate, not bit-exact equality, because GPU nondeterminism produces a small bounded number of off-by-an-LSB frames that should not block a promotion that is otherwise deterministic.

Watermark-robustness — at least one provenance signal per channel#

evaluateAllChannels in libs/yemaya/living-scenes-runtime/src/provenance/channel-robustness/channel-eval.ts runs the watermark-survival gate over the seven share channels in SHARE_CHANNELS: telegram, whatsapp, twitter-x, instagram, discord, email, generic-web. For each channel it counts fixtures where at least one of the three redundant signals survived the channel's transcode / strip behavior — visibleMark, audioWatermark, or c2paSidecar. A channel passes only when every fixture surfaced ≥ 1 signal (atLeastOneSignalSurvivedCount equals the fixture count), and allChannelsPass additionally requires that no channel is missing fixtures (missingChannels is empty). This is the eval-side companion to the four redundant provenance channels documented in Scene Safety, Determinism, Provenance, and Cue Privacy: the point of three signals plus a resolvable attestation page is that no single channel can strip provenance, and this gate proves it empirically per channel rather than asserting it.

The visible-mark side of that survival contract is real: buildAttestationUrl in provenance/visible/visible-mark.ts constructs the attestation page as ${origin}/scene/${encodeURIComponent(shortCode)} from a 6-character Crockford-style short-code (confusable 0/O/1/I/L removed) — keyed on the short-code, not an artifact <id>, and with the origin injected rather than a hardcoded oshun.app. Earlier doc drafts that wrote the attestation page as the literal oshun.app/scene/<id> were stale on both counts.

Cue-policy evals — fail-closed parsing + a precision/recall corpus#

The cue-policy gate has two halves, both real. The structural half is parseCue / sanitizeForCueParse in libs/yemaya/living-scenes-runtime/src/cues/cue-parser.ts: free text and ASR transcripts are sanitized (quotes, control characters, code-fence markers, and HTML tags stripped; whitespace collapsed; truncated to 240 chars) and then mapped to a small constrained verb vocabulary. Anything that is not a recognized verb returns verdict: 'unrecognised', and a voice cue below LOW_CONFIDENCE_THRESHOLD = 0.6 returns verdict: 'low-confidence' — so out-of-vocabulary or low-confidence cues fail closed and raw text never reaches the segment generator. That constrained-verb mapping is the prompt-injection defense, which is why the Cue verb mapping test below asserts exactly this fail-closed behavior.

The policy half is the redaction precision/recall evaluator. evaluateCorpus in libs/yemaya/living-scenes-runtime/src/cue-privacy/intent-redaction.ts runs the eight LILITH_REDACTION_CATEGORIES (named-person, deceased, medical-condition, abuse-disclosure, location-of-safety, minor-identity, financial-account, legal-status) — each a deterministic regex + replacement token — over a labeled fixture corpus and reports per-category and macro precision/recall. This is a deterministic pattern-matcher robust for the test taxonomy, not an ML classifier; the module header is explicit that real-world deployment will swap in a model-backed classifier with the same signature.

A note on which launch locales the cue-policy and crisis evals run over: the codebase carries two divergent launch-locale sets, and any cross-locale eval checklist must pick one and flag the divergence. The platform set OSHUN_LAUNCH_LOCALES (libs/oshun/i18n/src/index.ts) is eight locales — en-US, es-US, fr-FR, de-DE, ar, he, ja-JP, pt-BR. The Living-Scenes locale-parity set V1_LAUNCH_LOCALES (libs/isis/workflow-classes/src/living-scene/locale-parity.ts) is ten — en-US, en-GB, es-ES, es-419, fr-FR, de-DE, pt-BR, ja-JP, ar-EG, he-IL (with RTL_LOCALES = {ar-EG, he-IL}). The LS-scoped evals (cue policy, crisis parity, locale-parity fixtures) are authoritative over the ten-locale region-tagged set; treat the eight-locale i18n catalog as the platform string-coverage set, not the LS eval target. See Taxonomy, Localization, and Versioning for the broader localization model.

Crisis-detection evals — locale parity with a hard imminent floor#

checkLocaleParity in libs/yemaya/living-scenes-runtime/src/cues/crisis-classifier.ts is the scene-stream crisis gate: it requires the scene detector to be at parity with the conversational detector across the Lilith taxonomy. The taxonomy is the eight LILITH_CRISIS_LABELS — self-harm-imminent, self-harm-ideation, harm-to-others, severe-dissociation, panic-acute, overwhelm, distress, safe — and decideCrisisAction maps them to a severity of imminent (self-harm-imminent, harm-to-others), severe (self-harm-ideation, severe-dissociation, panic-acute), or moderate (overwhelm, distress). The parity gate is a LocaleParityReport that is atParity only when macroF1 clears the threshold and the imminentFalseNegativeRate is ≤ 0.02 — a hard 2% ceiling on missed imminent-class crises per locale, because a missed imminent signal is the most dangerous failure mode in the system. The classifier model is downstream; this module owns the decision shape, the taxonomy mapping, and the parity contract so any compatible classifier output can be gated against the same floor.

Tests — the contracts each gate is verified against#

The following are the V1 test contracts. Most are real, executed Vitest suites in libs/yemaya/living-scenes-runtime/src/; the BFF-level ones are Playwright e2e specs over apps/oshun/bff/src/routes/living-scenes.ts (a 1,871-line route).

• Score round-trip: serialize → store → re-render produces a pixel-equivalent video within tolerance on N approved nodes (the determinism harness over the round-tripped Score).
• Cue policy enforcement: voice/text cues that violate Lilith policy are rejected at the cue gate, never reach the segment generator, and trigger the soft-rejected-cue UX. Backed by parseCue's fail-closed unrecognised / low-confidence verdicts.
• Cue verb mapping: free text and ASR transcripts map correctly to the constrained verb vocabulary; out-of-vocabulary cues fail closed.
• Crisis-frame entry: a crisis-signal cue, a conversational signal, or a re-detection-sweep result triggers an immediate fade-to-still + plain operator voice + Iris memory-write suspension + safety-incident record. This is exactly the cross-surface cascade in libs/oshun/trust-safety/src/crisis/crisis-frame-cascade.ts: a single lilith.crisis_frame.activated event (LILITH_CRISIS_FRAME_ACTIVATED_EVENT) fans out to the five CRISIS_FRAME_SURFACES — psyche, lilith-video, tara, iris, assistant — with haltSynthesis and suspendMemoryWrites pinned true and non-overridable, and crisis-frame-worker.ts binds the real Redis bus. See Lilith Persona Policy and Iris Memory and Identity.
• Watermark survival: artifacts pushed through every supported share channel recover at least one provenance signal per channel (evaluateAllChannels, seven channels, ≥ 1-signal-survival pass).
• Determinism: same Score + Envelope ⇒ pixel-equivalence; an engine upgrade surfaces a "v2 available" notice; a rollback keeps pinned artifacts rendering under their pinned envelope.
• Shareability matrix: per-component restrictions are enforced at the share endpoint, fallback shares present the correct degradation, and crisis-touched artifacts are private-only regardless of other components — the resolveShareability matrix (nine components, four reach levels, minimum-reach resolution, crisis precedence), detailed in Keep, Share, Shareability, Takedown, and Lineage.
• Takedown cascade: voice retirement, source retraction, workflow-class freeze, persona termination, and account deletion all reach kept and shared artifacts within SLA; nightly sweeps verify compliance. The seven structured dispositions (DISPOSITION_KINDS) and their DispositionImpact records live in takedown/dispositions.ts; the revocation SLA is the concrete LIVING_SCENE_REVOCATION_SLA_SECONDS = 60.
• Tenant share policy: per-tenant share toggle, scope toggle, retention policy, and audit are enforced; ambiguous policy fails closed. In the BFF this is resolveTenantSharePolicy returning denied_tenant_internal_scope when a tenant-internal scope is exceeded; the deny-header path is gated behind OSHUN_LIVING_SCENES_HEADER_POLICY_OVERRIDES (honored only in non-production runtimes), and the living-scene-card e2e spec exercises it.
• Public viewer: noindex respected by default (LIVING_SCENE_DEFAULT_ROBOTS_META = 'noindex, nofollow, noarchive', flippable to LIVING_SCENE_INDEXABLE_ROBOTS_META = 'index, follow' on opt-in), rate-limit and abuse pipelines functional, and a per-page abuse report routing to Lilith. Retired offerings show the tombstone copy PUBLIC_VIEWER_TOMBSTONE_COPY = 'This offering has been retired.'. The public-scene-abuse-report and scene-viewer-deepening e2e specs cover these paths (per WALKTHROUGH/results/v1-triage-2026-06-23.md).
• Reduced-motion parity: every full-motion render has a reduced-motion render with equivalent narration and AD; reduced-motion is independently shareable.
• Accessibility: caption accuracy, AD coverage, screen-reader navigability, keyboard cue access, and OS-accessibility-setting respect.
• Backend swap: AAA-tier sessions can flip between chained-clip and phase-176-nous without breaking the customer surface; contemplative-tier sessions never resolve to phase-176-nous (the resolveBackend hard guard).
• Lineage and remix: replay-with-tweak from public artifacts produces correctly attributed children; takedown of a parent does not delete unrelated children; cross-user remix attribution is honored per consent. See Output Gallery, Lineage, Branch, and Replay.
• Latency budgets: P50/P95 frame latency, segment latency, and cue latency are met under nominal load and under backpressure (the degradation ladder above).

Content-addressed idempotency in the share path#

A subtle but real correctness property the e2e suite pins is that the keep/share endpoints in apps/oshun/bff/src/routes/living-scenes.ts are content-addressed: two identical shares of the same artifact at the same time mint the same shortCode / shareId, so a retried share is idempotent rather than creating a duplicate public link. This is why the scene-viewer-deepening and living-scene-card specs run with OSHUN_LIVING_SCENES_HEADER_POLICY_OVERRIDES=1 and an injected x-oshun-now clock: without a deterministic clock, two same-time shares land on different seconds, produce different short-codes, and break the idempotency assertion. The override is a test-only seam — it is ignored in production runtimes and falls back to the wall clock.

Related#

• Scene Safety, Determinism, Provenance, and Cue Privacy — the governance spine these gates verify: the PSE killer, crisis frame, determinism harness, and the four provenance channels.
• Keep, Share, Shareability, Takedown, and Lineage — the shareability matrix, takedown dispositions, and personal-artifact lifecycle the share and takedown tests exercise.
• Live Direction, Conductor Runtime, and Blend Kernel — the cue and blend pipeline whose latency budgets this page sets.
• Isis Generation Control — the release-gate dashboard that surfaces eval failures and blocks promotion.
• Generation Audience Tiers and Surface Boundaries — the entitlement tiers the backend resolver keys on.
• Living Scenes — Concept and Customer Promise — the customer promise these quality and accessibility bars protect.
• The features hub: ../features.md.