V1 Residual Audit — Slice 13: Foundations, Design System, Analytics/Observability, Testing/QA, Localization, Launch + Exit Criteria

Date: 2026-06-11 · Spec: V1/features.md lines 5983–6242 · Auditor scope: RESIDUAL after the 2026-06-10 ground-truth audit (all 57 backlog tasks landed by 2026-06-11).

Method: static source reading + grep only (no builds/tests run). Every claim below was verified against current code in this session; backlog task text was used only as a map, never as proof.

Part 1 — Findings#

F1. The C11 analytics loop is closed on paper only: the ONLY module wired to the real transport is itself orphaned; every LIVE customer telemetry module still discards#

Severity: P0-STRUCT

Evidence:

• apps/oshun/web/src/analytics/transport.ts:17 — createSharedAnalyticsSinks() (real createBufferedSink → POST /v1/analytics/events). Grep for consumers: createSharedAnalyticsSinks appears in exactly ONE non-test module — taraAnalytics.ts:394.
• taraAnalytics.ts itself has zero importers: grep for analytics/taraAnalytics, useTaraAnalytics, trackMeditationStarted, trackOnboardingCompleted across apps/oshun/web/src/{components,app,lib} → 0 hits outside the analytics dir.
• The telemetry modules that ARE imported by live components all still console-discard. Census of the 58 non-test modules in apps/oshun/web/src/analytics/:
  • 11 modules have component importers. 10 of them use a local console.info('[oshun-analytics]', …) sink with the literal comment "Temporary local sink until production telemetry transport is connected": activityReentryTelemetry.ts:94-96, domainLaunchTelemetry.ts:16-19, homeContinuationTelemetry.ts:159-160, librarySaveTelemetry.ts:26-28 (7 importers), nisabaStudyTelemetry.ts:139-140, publicAuthFunnelTelemetry.ts:122-123, recommendationTelemetry.ts:123-124, searchResultTelemetry.ts:61-62, shellNavigationTelemetry.ts:6-9, veritasWorkspaceTelemetry.ts:13.
  • The 11th, studioWorkspaceMountTelemetry.ts:67-159, transmits via its own pre-C11 BffTelemetrySink to /v1/studio/telemetry/workspace-events (route real: apps/oshun/bff/src/routes/studio-workspace-telemetry.ts:37).

Spec promise: features.md 6113-6118 — customer event taxonomy for activation, retention, domain navigation, home, search, library, onboarding; assistant taxonomy.

What the code actually does: The BFF ingest (apps/oshun/bff/src/analytics/events-ingest.ts) and the shared transport are real, well-built code — but in the running product, zero events from the customer activation/retention/search/library/auth-funnel taxonomy ever reach them. The only web telemetry that transmits anywhere is the studio workspace-mount module (3 event kinds, to a different endpoint). The 06-10 headline "68/69 web telemetry modules never transmit" is now "57/58 never transmit, and the 1 flipped module has no callers."

Fix sketch: Swap the 10 live modules' local console sinks for createSharedAnalyticsSinks() (one-line change each — the shapes already match), and either wire useTaraAnalytics() into the Tara session player/onboarding components or delete taraAnalytics.

F2. 28 telemetry modules route through a singleton that is never initialized → silent no-op discard in every environment#

Severity: P1

Evidence:

• libs/oshun/analytics/src/singleton.ts:30-36,52-56 — getOshunAnalytics() returns lazyNoOpClient() whose sink body is // Silently discard when analytics is not yet initialized. unless initOshunAnalytics() was called.
• Grep for initOshunAnalytics across apps/ and libs/ (excluding tests): the ONLY hit is the definition in singleton.ts itself. No app ever initializes it.
• 28 web modules depend on it (studioPerformanceBudgetsTelemetry.ts:62-74, studioExperimentationFeatureFlagsTelemetry.ts, studioObservabilityOperationalDashboardsTelemetry.ts, studioAccessibilityGovernanceTelemetry.ts, all studioBellona* bridge telemetry, etc.). Their fallback (studioPerformanceBudgetsTelemetry.ts:49-55) only console-warns in development if the dynamic import fails — in production every event is swallowed without a trace.

Spec promise: features.md 6113-6120 (admin + studio event taxonomies).

What the code actually does: A worse variant of the console-sink modules: not even console output. These are silent-discard façades in dev AND prod.

Fix sketch: Call initOshunAnalytics({ context, sinks: createSharedAnalyticsSinks() }) once in lib/providers.tsx (next to ExperimentationBootstrap). One call un-façades all 28 modules. (Most are also orphaned — see F11 — so triage which to keep first.)

F3. The entire apps/oshun/web/src/observability/ directory is dead: never initialized, and posts to five BFF routes that do not exist#

Severity: P1

Evidence:

• performance-telemetry.ts:138,177,217 posts to /v1/web/telemetry/startup, /v1/web/telemetry/route-transition, /v1/web/telemetry/diagnostics; error-monitoring.ts:291,309 posts to /v1/web/errors, /v1/web/vitals; crash-reporting.ts:203,219 posts to /v1/web/crashes, /v1/web/diagnostics.
• Grep v1/web across apps/oshun/bff/src: zero route registrations. All seven endpoints 404.
• Grep for initPerformanceTelemetry, initWebErrorMonitoring, initializeCrashReporting in web src/: zero callers (web has no instrumentation.ts; app/error.tsx:26 merely mentions the module in a comment).

Spec promise: features.md 6121-6126 (service-health dashboards, structured tracing) and 6240-6242 (post-deploy monitors as a launch gate).

What the code actually does: Web crash reporting, error monitoring, web-vitals reporting, and startup/route-transition performance telemetry are all non-functional twice over: nothing starts them, and their fetch targets don't exist. The catch {} in each postTelemetry would swallow the 404s even if they were started.

Fix sketch: Add BFF receiver routes (or point them at the C11 ingest envelope), then initialize from a client-root effect. Or delete the directory and route errors through the C11 ingest.

F4. Mobile crash reporting IS initialized — and posts every crash into a 404#

Severity: P1

Evidence:

• apps/oshun/mobile/app/_layout.tsx:37,130 calls initializeCrashReporting(...).
• apps/oshun/mobile/src/observability/crash-reporting.ts:290,309 posts to /v1/mobile/crashes and /v1/mobile/diagnostics.
• Grep mobile/crashes|mobile/diagnostics in apps/oshun/bff/src: zero routes.

Spec promise: features.md 6148-6150 (mobile coverage incl. retry/diagnostics), 6240-6242.

What the code actually does: Real crashes on real devices are serialized, POSTed, 404'd, and silently dropped. This is façade telemetry on the live mobile path — the one place a crash report matters most.

Fix sketch: Register POST /v1/mobile/crashes + /v1/mobile/diagnostics in the BFF backed by a durable snapshot store (the C11 ingest pattern is directly reusable), with an operator read.

F5. Localization is a façade: 8 launch languages and a polished switcher over a hardcoded-en, zero-consumer message system#

Severity: P1 (with a UX-dishonesty component)

Evidence:

• libs/oshun/i18n/src/index.ts:40-50,99 — OSHUN_LAUNCH_LANGUAGE_PREFERENCES = en, es, fr, de, ar, he, ja, pt; RTL_LOCALES = ar, he.
• apps/oshun/web/src/i18n/messages/ contains ONLY en/ — 5 namespace files, 208 lines total for a 684-route app.
• apps/oshun/web/src/i18n/request.ts:5 — const locale = defaultLocale; — the next-intl request config hardcodes en; the user's stored preference is never consulted for messages.
• Grep useTranslations|getTranslations|NextIntlClientProvider across web src/ excluding src/i18n/: zero hits. Not one component renders a translated string.
• components/LanguageSwitcher.tsx:26-35,128-139 — a full dropdown with flags for all 8 locales that persists the selection (savePreferences({ language: loc })) and flips document direction (locale-store.ts:8 applyLocaleToDocument) — while every visible string stays English.

Spec promise: features.md 6181-6184 — "Locale architecture, externalized strings, …, RTL and text-expansion QA, and launch locale coverage for customer and admin journeys."

What the code actually does: A member who selects Arabic gets an RTL document direction over 100% English hardcoded strings. The locale machinery (config, store, formatting, preference sync) is real; the localization itself does not exist, and the switcher implies it does. No deferral for i18n is recorded in the 06-10 audit backlog.

Fix sketch: Either (a) honest-scope it: ship the switcher as "English (more languages coming)" with non-en options disabled, or (b) thread the stored locale into request.ts, externalize shell-level strings, and produce real catalogs for the declared launch locales.

F6. The public /status page renders hardcoded specimen data while the live E8 status API has zero consumers#

Severity: P1

Evidence:

• apps/oshun/web/src/app/status/page.tsx:3,13 renders StatusPage from components/lilith/system-pages.tsx, whose data is the hardcoded STATUS_COMPONENTS array (system-pages.tsx:545), fixture incidents (:564-566 — "Fixture · Tara · CDN edge…"), and a pill that literally reads Specimen · fixture data (:601).
• The real route GET /v1/status (apps/oshun/bff/src/routes/customer-communications.ts:149, E8: live-derived component states, operator overrides win) has zero consumers: grep /v1/status across all apps excluding the BFF → only an unrelated lilith/svc-analytics doc route.

Spec promise: features.md 6230-6231 (exit criterion: "the public status page [is] ready for launch") and 6031-6032 (status plumbing "bound to component health checks").

What the code actually does: The page is honest (it labels itself specimen), so this is not result-faking — but the launch-facing status page shows fake components/incidents and the genuinely live status derivation built for E8 is a dead route.

Fix sketch: Make StatusPage fetch GET /v1/status (server component fetch is fine — the route is public) and render the real components/incidents/maintenance; keep the specimen behind a dev query param like the banner catalog.

F7. Outbound webhook plumbing: authored webhooks are never dispatched by any real event#

Severity: P1

Evidence:

• The only "delivery" code is apps/oshun/bff/src/studio/webhook-delivery-store.ts:1-12 — a pure, deterministic classifier of already-known response statuses ("0 Date.now/random"), feeding the admin console route. It performs no HTTP.
• Grep dispatchWebhook|deliverWebhook|webhook.*fetch( across apps/oshun/bff/src: no outbound dispatcher exists. Webhook/API-key records live in the integrations registry stores (admin/admin-integrations-registry-store.ts) and tenant console, with no event→delivery wire.

Spec promise: features.md 6016-6018 — "Webhook and event-bus plumbing for outbound notifications, with topic registry, schema versioning, signing, retry/backoff, dead-letter routing, replay, and delivery audit."

What the code actually does: Tenant/operator-authored webhooks are stored configuration that no runtime event ever fires. This is the surviving half of the 06-10 "management without enforcement" headline (#5) that the backlog never tasked.

Fix sketch: A small dispatcher worker: subscribe to the existing in-BFF event seams (feedback intake, generation release, incident publish), sign + POST to registered endpoints, and feed real outcomes through the existing classifier/backoff store.

F8. Stored API keys are never validated on any inbound request#

Severity: P1

Evidence:

• API keys exist only as managed records: apps/oshun/bff/src/admin/admin-integrations-registry-store.ts / routes/admin-integrations-registry.ts (all handlers are admin-console CRUD, preHandlers [abuseProtection, authProtection]).
• Grep x-api-key|validateApiKey|apiKeyAuth across apps/oshun/bff/src (and middleware/): zero. No request path on the BFF accepts or checks an API key.

Spec promise: features.md 6012-6015 — "scoped tokens, …, rate limits, per-key quotas, audit" for the public API platform; exit criterion 6220-6223 (Tenant Console "API keys").

What the code actually does: The tenant-console "API keys" surface mints credentials that grant nothing and guard nothing — stored-config-never-enforced. (Unchanged from the 06-10 headline; no backlog task covered it.)

Fix sketch: Either remove/disable the key-issuing surface for V1 (honest), or add an x-api-key preHandler that resolves keys to tenant+scopes for a defined public-API route subset.

F9. Contracts still sit beside the routes, not under them#

Severity: P1

Evidence (counts from this session):

• apps/oshun/bff/src/routes/ has 487 route files; 31 import @oshun/contracts; 44 call .parse(/.safeParse(; 12 import zod directly. The overwhelming majority hand-roll typeof/shape checks (e.g. routes/feature-flags.ts:107-224 — 120 lines of manual parsing) or validate nothing.
• Even the new C11 ingest hand-rolls (analytics/events-ingest.ts:99-119 — typeof checks per field) although the analytics library ships validateEventEnvelope (libs/oshun/analytics/src/validation.ts:108) and a full event-name taxonomy it never consults.
• libs/contracts/src/ is a large, real schema tree (common/, events/, per-domain) — with contracts.spec.ts style self-tests but thin route-side enforcement.

Spec promise: features.md 6058-6059 — "Object-property authorization, excessive-data-exposure, and mass-assignment protections enforced through canonical Zod request/response shapes"; 6002-6005 (response normalization, contract tests).

What the code actually does: The contract library exists and drifts checked against itself, but ~94% of route files don't bind it to their payloads. Hand-rolled parsing is mostly careful, but response-shape (excessive-data-exposure) protection is uniformly absent as a systematic guarantee.

Fix sketch: Adopt fastify-type-provider-zod (or a thin parseWith(schema) helper) and migrate routes in tranches starting with mutation routes; add a lint rule banning untyped request.body access in new routes.

F10. No OpenTelemetry / distributed tracing anywhere on the Oshun runtime path; tracing manifest has zero consumers#

Severity: P2

Evidence:

• Grep opentelemetry in apps/oshun/bff (src + package.json): zero. OTel deps exist only in apps/lilith/svc-media/package.json:31-37 and libs/iris/core (its own tracing module) — neither is imported by the Oshun BFF or web.
• The only trace awareness in the BFF is middleware/residency-guard.ts:200-202 reading an inbound traceparent header into an audit field. Logging is the Fastify default (app.ts:557 logger: true).
• libs/oshun/analytics/src/tracing-manifest.ts (plus alerts-manifest.ts, incident-ownership-manifest.ts): grep across the repo — zero consumers outside the library.

Spec promise: features.md 6125-6126 — "Structured tracing across shell, BFF, Sophia, Iris, Psyche, Lilith, Isis, Metis, review, support, and admin workflows."

What the code actually does: Request-scoped pino logs + a Prometheus request counter (C11, real — app.ts:706-720). No spans, no propagation, no exporter. The tracing manifest is documentation-as-code with no enforcement seam (unlike the journey-coverage manifests, which DO verify spec files exist — __tests__/web-playwright-journey-coverage.test.ts:32).

Fix sketch: @opentelemetry/sdk-trace-node + fastify instrumentation in the BFF behind an env-gated exporter; propagate traceparent to domain-service adapters. Or record an explicit V1.x deferral — today the spec sentence is simply unmet.

F11. 47 of 58 web telemetry modules are orphaned dead code; dashboard/evaluation/experimentation manifests have zero consumers#

Severity: P2

Evidence:

• Importer census (this session): of 58 non-test modules in apps/oshun/web/src/analytics/, 47 have zero importers outside the analytics dir (all studio*Telemetry flow-span modules, taraAnalytics, taraExperiments). 11 are imported (F1 lists them).
• libs/oshun/analytics/src/dashboards-{customer-kpi,nisaba,assistant,service-health,queue-health, operational-readiness,satisfaction}.ts, evaluation-manifest.ts, experimentation-manifest.ts: zero consumers anywhere outside the library (grep across apps/ + libs/). The only symbols any app imports from @oshun/analytics are the client/sink/singleton ones.
• The operator read GET /v1/admin/analytics/events (events-ingest.ts:127) also has no UI consumer — no admin console fetches it.

Spec promise: features.md 6121-6124 — "Dashboards for activation, retention, cross-domain use, … service health, queue health, … launch readiness."

What the code actually does: The dashboard layer of the spec exists as typed manifest objects that nothing renders, queries, or enforces. There is no activation/retention dashboard a human can open.

Fix sketch: One admin "Analytics" console reading /v1/admin/analytics/events facets + /metrics-derived route health would make the C11 ingest observable; delete or enforcement-test the manifests like the journey-coverage ones.

F12. Feature-flag/experimentation machinery runs end-to-end but gates nothing; all experiment definitions are at 0% rollout#

Severity: P2

Evidence:

• Server: apps/oshun/bff/src/feature-flags/tenant-feature-flags.ts:129 defines real flags (shellDomainConfiguration, domainSwitcher, quickActionsTray, homeLayoutV2, …) with a real deterministic bucketing evaluator; the experiment-bearing definitions carry rolloutPercentage: 0 at lines 197, 213, 229, 242, 258. Env overrides exist (OSHUN_TENANT_FEATURE_FLAG_OVERRIDES, line 319). Route routes/feature-flags.ts:63 is live.
• Client: experimentation/bootstrap.tsx:73-93 initializes + refreshes flags on every pathname/ profile change from lib/providers.tsx:12. Hooks exist (experimentation/hooks.ts:20,37).
• Consumers: grep useFeatureFlag|useFeatureFlagValue outside src/experimentation/: zero. Not one component branches on a flag.
• analytics/taraExperiments.tsx (FNV-1a deterministic assignment, exposure/conversion tracking, useTaraExperiment): zero importers.

Spec promise: features.md 6127-6128 — "Experiment support for recommendation ranking, assistant presentation, disclosure presentation, and home/dashboard composition"; 6026-6027 tenant-aware flags/experiment scoping; 6051 "validated configuration for feature flags … experiment guardrails."

What the code actually does: A flag evaluation round-trip fires on every page navigation (a real network cost) and the result is read by nobody. Experimentation capability: present; experimentation: nonexistent. Not result-faking (nothing claims an experiment ran) — orphaned machinery + wasted hot path.

Fix sketch: Wire homeLayoutV2/domainSwitcher into their components as the first real gates, or stop bootstrapping flags on every navigation until a consumer exists.

F13. GET /metrics is unauthenticated and enumerates the full route inventory#

Severity: P2

Evidence: apps/oshun/bff/src/app.ts:717-720 — app.get('/metrics', …) with no preHandler; prometheusMetrics() emits per-route counters (route template names incl. /v1/admin/* paths, status codes, latencies) for anything that has handled a request.

Spec promise: features.md 6071-6072 — "API inventory and shadow-endpoint discovery checks for deprecated, preview, internal, and admin-only routes" (the spec treats route inventory as sensitive); 6068-6070 resource-consumption controls.

What the code actually does: Any anonymous caller can scrape admin route names, traffic volumes, and error rates. Acceptable on a private network; this BFF also serves the public web.

Fix sketch: Gate /metrics behind the observability admin scope or a static bearer from env (OSHUN_METRICS_TOKEN), defaulting closed in production.

F14. Anonymous flooding can evict the entire 5000-event analytics retention ring#

Severity: P2

Evidence: apps/oshun/bff/src/analytics/events-ingest.ts:87-123 — POST /v1/analytics/events is sessionless by design (documented), accepts 100 events/request, unshifts and prunes to RETENTION_MAX = 5000. Only the generic abuse-protection limiter stands between an attacker and 50 requests that erase all real telemetry (and the durable snapshot persists the junk).

Spec promise: features.md 6068-6070 — resource-consumption and abuse controls for analytics APIs.

What the code actually does: Honest short-retention design, but eviction-by-write means the operator read can be blinded cheaply. Low stakes today (nothing consumes the data — F11), higher once dashboards exist.

Fix sketch: Partition retention per platform/origin, or require a session for non-anonymous event names, or rate-limit by event volume not request count.

F15. OpenAPI documents ~91 paths of a 487-route-file surface#

Severity: P2

Evidence: apps/oshun/bff/openapi/oshun-bff.openapi.yaml (25,612 lines, ~91 /v1/ paths). The contract tests check the documented subset exists at runtime (__tests__/contract/openapi-runtime-drift.test.ts) and that a hand-picked "critical" list is documented (openapi-coverage.test.ts:113-126) — there is no inverse check, so hundreds of live routes (incl. the new C11/C8/E8 routes) are undocumented and undetectable as drift.

Spec promise: features.md 6002-6005 — "OpenAPI coverage … schema drift checks."

What the code actually does: One-directional drift checking over a fraction of the surface.

Fix sketch: Add the inverse assertion (enumerate app routes, allowlist intentionally undocumented ones) so new routes must either be documented or explicitly waived.

F16. BFF telemetry/ modules (search + recommendation) — verify their feed#

Severity: P2 (downgraded observation)

Evidence: apps/oshun/bff/src/telemetry/{search-telemetry.ts,cross-domain-recommendation-telemetry.ts} exist server-side, but their client-side feeders (searchResultTelemetry, recommendationTelemetry) are console-only (F1) — the spec's "recommendation signal ingestion" (6232-6233) therefore has no live client signal. The recommendation scoring itself is real (recommendations/scoring.ts consumed by routes/recommendations.ts), and search offline evals exist (search/offline-eval-route.ts, libs/oshun/search-discovery/src/evals/offline-evals.ts).

Fix sketch: Falls out of F1 — flipping searchResultTelemetry/recommendationTelemetry to the shared transport gives the ingestion leg its first real signal.

Part 2 — What is genuinely solid in this slice (verified, not assumed)#

• C11 server side is real: durable capped ingest with prune-on-write (events-ingest.ts:64-67), scope-gated operator read with facets, and a genuinely registered BffMetricsCollector — app.ts:707-716 records every response via an onResponse hook and /metrics serves computed Prometheus text. The library BufferedAnalyticsSink (libs/oshun/analytics/src/buffered-sink.ts) is a real batching/retry/backoff HTTP sink.
• Accessibility posture is strong: e2e/support/accessibility.ts runs AxeBuilder with WCAG 2.0/2.1/2.2 A+AA tags; 10 spec files consume it (shell-route-accessibility.spec.ts hard-fails on any violation at lines 142-143; wcag-aa-signoff-*, assistant/nisaba/search/tara a11y specs); prefers-reduced-motion honored in globals.css and player/assistant components; a11y-pa11y-nightly.yml + accessibility.yml in CI.
• Performance budgets are enforced: apps/oshun/web/lighthouserc.cjs asserts numeric budgets (perf ≥0.88, FCP ≤1800ms, LCP ≤2500ms, TBT ≤250ms, CLS ≤0.1) over authenticated critical routes, with lighthouse-budget.json route budgets, wired to PR/push in oshun-web-lighthouse.yml (+ admin/metis variants, oshun-web-bundle-monitor.yml).
• E7 design-system decision is real and enforced: src/design-system/__tests__/lilith-palette-regression.test.ts literally diffs the TS palette hexes against lilith.css custom properties.
• e2e/CI discipline holds: 254 customer + 76 admin Playwright specs + mobile Maestro flows; journey-coverage manifests are enforcement-tested against real spec files (web-playwright-journey-coverage.test.ts:32 existsSync per entry); CodeQL, scheduled authenticated DAST (security-dast.yml, daily vs staging), SBOM/signing, secret scanning, stub-indicator-scan.yml, test-coherence-check.yml all present as real workflows.
• Docs/runbooks/launch material exist as deliverables: docs/runbooks/ covers the spec's named runbooks (shell-outage, grounding-failure, assistant-failure, provider-failover, moderation-surge, privacy-incident, model-workflow-rollback, persona-rollback, provenance-failure); docs/launch/ has go-no-go, canary-analysis, post-deploy-monitoring, critical-journeys, locale-coverage, exit-criteria-signoff, private-beta, dogfood-drill; docs/training/ exists. (Documents verified present; their operational truth is inherently a deploy-time question.)
• Feature-flag evaluator and kill-switch enforcement are real server code (F12 is about consumers, not fakery): agentic/runs-route.ts maps operator-armed kill switches into the execution guard (A5), and the flag evaluator does deterministic bucketing.

Part 3 — V1 Exit Criteria scorecard (features.md 6199-6242)#

Verdicts: MET / PARTIAL / UNMET, each from code read in this session plus the 06-10 audit's completed-task evidence where cross-slice.

Scorecard totals: 3 MET · 10 PARTIAL · 4 UNMET (criteria 7, 12, 15, 17-observability-leg; #17 counted PARTIAL overall). Strictly: MET 3, PARTIAL 11, UNMET 3 if #17 is taken as one PARTIAL.

Severity count table#

Total: 16 findings (1 P0-STRUCT, 8 P1, 7 P2, 1 UX-component).