Observability, Design System, Testing, and Performance

This page covers the platform-quality spine that sits underneath every Oshun V1 surface: how the system is observed and evaluated, how its visual and interaction language is encoded as code, how it is tested, and how its performance is budgeted and enforced. These concerns are cross-cutting — they serve every customer-facing domain (Tara, Veritas, Nyx, Arete, Nisaba, Metis), the admin/Studio surfaces, and the substrates (Sophia, Iris, Psyche, Lilith, Isis, Aje) — and they are where "is V1 ready to ship?" is actually answered. For the most part this is implemented, tested code, not a spec: the i18n catalog, analytics taxonomies and manifests, design tokens, the UI component library, and the design-language audit modules are all real, sizeable, exercised libraries. What remains genuinely operational — the launch drills, pen-test signoff, and go/no-go gates — is described candidly as process in Content, Localization, Documentation, and Launch Readiness.

Product scope for the design system lives in V1/features.md § Design System and Experience Quality; testing/QA/performance scope in V1/features.md § Testing, QA, Accessibility, Performance, and Security. Backlog: §29 (design system), §31 (testing/QA), plus the observability and evaluation items in §30. The hub is ../ARCHITECTURE.md.

Observability, Evaluation, and Release Gates#

Observability in V1 is the standard three-pillar setup — traces, logs, metrics — wired through shared libraries so that every service emits the same shapes, and backed by named, machine-readable evaluation and release manifests so that "ready to promote" is a checkable fact rather than a judgment call.

Tracing, logging, metrics#

• Structured tracing spans the shell, the BFF, and the substrates — Sophia, Iris, Psyche, Lilith, Isis — plus the domain services. Spans export over OTLP via OTEL_EXPORTER_OTLP_ENDPOINT, and the trace UI is Jaeger.
• Logging flows through @oshun/logging (package at libs/shared/logging/), which is a Pino logger (pino ^8.18.0) with the surrounding middleware, sampling, and transport plumbing — its source carries logger.ts, events.ts, and per-module tests (logger.spec.ts).
• Metrics flow through @oshun/metrics (libs/shared/metrics/), which wraps prom-client (prom-client via the pnpm catalog) behind a shared registry — registry.ts plus a metrics server.ts expose the Prometheus scrape surface, and helpers.ts standardizes counter/histogram construction. Dashboards land in Grafana.

The point of routing everything through this shared plumbing is uniformity: a trace from Sophia's retrieval path and a trace from the assistant's tool loop use the same field names, so a single Grafana/Jaeger query works across substrates, and an SLO breach alert fires off one canonical metric name rather than a per-team dialect.

Evaluation pipeline — evaluation-manifest.ts#

The evaluation discipline is encoded concretely in libs/oshun/analytics/src/evaluation-manifest.ts. It is not prose: it is a set of exported constants the CI/CD release gates consume directly.

OSHUN_V1_EVALUATION_SCOPES enumerates the twelve things V1 evaluates, so no launching surface can quietly ship without a scope:

OSHUN_V1_EVALUATION_PIPELINE_STAGES defines the seven ordered stages every eval run walks through:

select_dataset → run_inference → apply_metrics → compare_to_baseline
  → check_thresholds → publish_summary → attach_to_release_readiness

The final stage — attach_to_release_readiness — is the load-bearing one: it binds an eval result to the release record so that a green eval is evidence on the release, not a transient log line. The companion constants OSHUN_V1_BENCHMARK_DATASETS (the dataset registry, queryable by scope and by id via the exported helpers), OSHUN_V1_PROMOTION_THRESHOLDS (per-scope bars a candidate must clear), and OSHUN_V1_REGRESSION_BLOCKING_RULES (the rules that fail a build closed when a metric regresses) complete the loop. The file header ties these to the backlog ids V1-OBS-020…V1-OBS-023.

Experimentation and champion–challenger#

Rollouts for the assistant, generation, and search use champion–challenger promotion: a challenger only takes over once it clears a passing eval against a gold set. The experiment shape is the exported ExperimentDefinition in libs/oshun/analytics/src/experimentation-manifest.ts, which carries arms (each with an allocation share that the arms must sum to 100), successMetrics, guardrails, and an allocationStrategy (e.g. random_stratified). The live catalog OSHUN_V1_EXPERIMENTS instantiates these definitions, so the experimentation surface is data, not an ad-hoc dashboard. Living Scenes carry their own eval sets and release gates, described in V1/features.md § Eval Sets, Release Gates, and Rollback.

The broader analytics package#

libs/oshun/analytics/src/index.ts re-exports roughly three dozen V1 modules. Beyond evaluation and experimentation, the V1-scoped set includes the customer event taxonomy, admin-taxonomy, assistant-taxonomy, flows-taxonomy, release-taxonomy, a tracing-manifest, an alerts-manifest, an incident-ownership-manifest, the QA coverage matrices (qa-coverage-matrices, qa-governance-and-enforcement), the CI-coverage manifests for web Playwright (web-playwright-ci-coverage), mobile e2e (mobile-e2e-ci-coverage), and accessibility/Lighthouse (accessibility-and-lighthouse-ci-coverage), plus the signoff/readiness manifests v1-31-testing-qa-signoff, v1-launch-readiness-manifest, v1-security-readiness-manifest, v1-release-and-exit-criteria-manifest, v1-33-launch-gate-signoff, and v1-33-launch-readiness-evidence. Note that the same package also still carries a large block of V3-era budget manifests (v3-ue-desktop-fps-budget, v3-ue-mobile-fps-budget, v3-ue-vr-fps-budget, v3-pixel-streaming-rtt-budget, v3-voice-latency-budget, v3-music-sync-drift-budget, v3-cold-join-budget, and more) — these are present in the tree but are out of V1 design-system scope; do not read them as V1 surfaces.

Customer event taxonomy — customer-taxonomy.ts#

The customer analytics contract is concrete and named in libs/oshun/analytics/src/customer-taxonomy.ts. OSHUN_V1_CUSTOMER_EVENT_FAMILIES is the closed set of eighteen event families:

lifecycle, auth, onboarding, profile, navigation, continuity, engagement, value_action, search, recommendation, notification, study, performance, error, privacy, commerce, accessibility, preferences.

OSHUN_V1_CUSTOMER_EVENT_NAMESPACES is the seven-namespace partition that scopes events to the platform shell and each consuming domain: oshun, tara, veritas, nyx, arete, nisaba, metis. The full event definitions live in OSHUN_V1_CUSTOMER_EVENT_TAXONOMY (each carrying legacyNames so older event names migrate cleanly), the canonical KPI subset is OSHUN_V1_CANONICAL_KPI_EVENT_NAMES, and resolveLegacyEventName() resolves an old name to its canonical form. This is the machinery behind the "activation, retention, value-action" KPIs — but spelled out as a typed, test-locked taxonomy rather than a loose convention.

Design System and Experience Quality#

The design system is encoded as code, with a single source of truth for tokens, a real React component library, and a separate suite of design-language audit modules that check the system against itself.

Design tokens — libs/oshun/design-tokens/#

The canonical token source is libs/oshun/design-tokens/src/tokens.ts. It is worth being precise about what the canonical surface actually is, because the older architecture text had it inverted.

The canonical V1 color system is a cool, dark ink/fog + aqua/amber palette, not "cream-paper + terracotta". oshunTokenSchema.color groups the scales as:

The default theme is dark: defaultOshunTheme = oshunThemes.dark. The oshunThemes map is keyed by OshunThemeName — 'light', 'dark', 'highContrastLight', 'highContrastDark' — and getSemanticTokens() / createCssVariables() default to that dark theme. The foundation bundle is versioned: OSHUN_V1_FOUNDATION_TOKEN_BUNDLE_ID = 'oshun-v1-foundation' at OSHUN_V1_FOUNDATION_TOKEN_BUNDLE_VERSION = '1.0.0'.

The cream-paper-and-ink + terracotta language people remember is the Lilith register (the "manuscript palette"), and it lives specifically inside the color.domain block of oshunTokenSchema (around line 1103), where each customer domain is mapped to a manuscript hue:

So the relationship is the reverse of the old claim: the ink/fog + aqua/amber dark system is the canonical surface, and the manuscript palette is the domain accent register applied at the persona-policy boundary (see Lilith — Contemplative Policy Substrate). The in-code comment is explicit that these domain accents are "mapped into the Lilith register … so the sidebar reads as the same cream-paper-and-ink language as the rest of the surface" — i.e. the manuscript feel is applied to the canonical system, not the other way around.

Tokens are not only colors. The same file exports the spacing scale (oshunSpacingScale), oshunFontWeights, the oshunTypographyRamp, the motion system (oshunMotionDurationScale, oshunMotionTokens), elevation (oshunElevationTokens), and the domain-accent behavior rules (oshunDomainAccents, oshunDomainAccentBehavior — which encodes that success/warning/danger/info always outrank domain identity for critical meaning). A further family of behavioral token sets encodes how trust-bearing UI must behave: oshunGroundedEvidenceUiBehavior, oshunDisclosureUiBehavior, oshunAssistantPersonaSwitchingUiBehavior, and oshunTrustSignalUiBehavior. These are the policy-as-tokens that keep, for example, an AI-disclosure chip from ever being styled like a celebratory success badge. Mobile derives the same tokens through @oshun/design-tokens rather than maintaining a parallel palette.

Component library — libs/oshun/ui/#

libs/oshun/ui/src/index.ts re-exports three subpackages — components/, motion/, and theme/. The component surface is concrete: the components/ directory ships roughly twenty-one React components — ActionSheet, Badge, Banner, BottomNav, BottomSheet, Box, Button, Card, Chip, CommandPalette, DomainSwitcher, EmptyState, ErrorState, IconButton, ListRow, Modal, Stack, StatTile, Text, Toast, and TopBar — grouped (data-display, feedback, navigation, overlays, primitives), each with their own .stories.tsx and .test.tsx, and an accessibility-snapshots.test.tsx that locks the a11y behavior of the whole set. These are the shared components consumed by every Oshun web/admin surface, so a DomainSwitcher or CommandPalette looks and behaves identically across domains.

Information architecture#

Shell ordering is Tara-led — the registry's OSHUN_SHELL_PRIMARY_DOMAIN places Tara at the center — and domain switching is a single gesture: one tap on mobile via DomainSwitcher/BottomNav, one keystroke (Cmd-K) on web via CommandPalette. The responsive grid pairs a desktop multi-panel workspace (apps/oshun/web/src/app/(workspace)/) with a one-handed mobile IA; both pull their breakpoints from the design tokens rather than ad-hoc media queries.

Accessibility as a baseline#

WCAG 2.1 AA is the V1 minimum, not an aspiration: contrast, reduced-motion variants, large-text variants, screen-reader controls, and keyboard-only flows. The token system carries highContrastLight/highContrastDark themes for the contrast case, and the design-token motion set carries the reduced-motion path. Tara audio and Living Scenes carry transcripts, captions, and alternative-modality variants; Living Scenes ship reduced-motion renders per V1/features.md § Reduced-Motion and Accessibility Renders.

Design-language audit modules — libs/oshun/design-language/#

A frequently-missed part of the system is that the design language is audited by code. libs/oshun/design-language/src/ ships a suite of modules, each with a matching .test.ts:

The catalog is the strongest of these. component-catalog.ts exports OSHUN_COMPONENT_CATALOG_VERSION = 'component-catalog-v1', OSHUN_COMPONENT_CATALOG (every cataloged component, each carrying its requirement, group, canonicalExport, implementationPath, testPath, customerShell/adminShell flags, and requiredA11yChecks), and OSHUN_REQUIRED_COMPONENT_REQUIREMENTS — the 22 requirements that must be covered: domain-switcher, continuity-card, evidence-card, evidence-sidebar, claim-item, source-item, passage-item, notebook, collection, assistant-disclosure, memory-state, provenance-badge, admin-queue, admin-filter, incident-severity, sla-timer, review-decision, policy-diff, avatar, voice-pack, rights-summary, and license-summary. The exported coverage function checks the catalog against this required set and reports totalRequirements/coveredRequirements, so a missing trust-bearing component (say, provenance-badge) fails the audit rather than slipping out silently.

Testing, Accessibility, and Performance#

Unit, integration, and e2e#

Unit and integration tests use Vitest — and per CLAUDE.md, the way to run a single suite while Nx is degraded by worktrees is npx vitest run path/to/test.ts from the library directory. E2E is Playwright across customer web, admin web, and Studio subroutes; keyboard-only journeys, visual regression, and resilience coverage are launch-gating, and the expected coverage is itself recorded as data in the analytics package's web-playwright-ci-coverage, mobile-e2e-ci-coverage, and the journey-coverage manifests. Accessibility is automated with @axe-core/playwright and backed by manual audits against WCAG 2.1 AA.

Honest status: the coverage and budget manifests exist and are test-locked, but this audit verified the files, not a live green Playwright/Lighthouse/axe run. Whether those checks currently pass on main is an operational fact, not one provable from the source tree alone.

Performance budgets — apps/oshun/web/lighthouse-budget.json#

Customer-web performance is enforced by apps/oshun/web/lighthouse-budget.json, a per-route Lighthouse budgets file. The root /* budget is:

Resource budgets cap the page at 450 KB total / 400 KB script (with 55 KB stylesheet, 90 KB image, 32 KB font, and zero third-party resources). Hot routes tighten the bars further — /search drops to 360 KB total and 2300 ms LCP, and /domains/nisaba to 355 KB total, 2200 ms LCP, and 0.05 CLS.

One correction to the older docs: the conceptual interaction target is INP, but the file does not encode an INP timing metric — Lighthouse budgets.json has no INP key. The actual enforced key is max-potential-fid at 200 ms, the older FID proxy. So "INP ≤ 200 ms" is the intent; "max-potential-fid ≤ 200" is what the file checks. LCP (2.5 s) and CLS (0.1) match the conceptual targets directly. Mobile budgets follow Expo's perf baseline plus battery, memory, and caching budgets per the device-matrix run.

Tenant isolation and the stub scan#

Tenant isolation is launch-gating: tests/security/tenant-isolation/ must pass, and any cross-tenant cache, feature-flag, or experiment leakage is treated as a P0. Separately, the repo's adversarial stub scan lives at scripts/stub-indicator-scan.sh — it runs at pre-commit on staged files and in CI on the whole PR diff, filtering .ts, .tsx, .rs, and .py, and exits non-zero on actionable stub markers (--mode=stubs), with --mode=random for Math.random() in implementation source and --mode=all for both. Its header ties it to CI jobs stub-indicator-scan and silent-random-scan, a nightly adversarial-grep-trend, and the pre-commit hook.

Eval sets and frontend verification#

Every V1-launching surface carries a per-surface eval harness (assistant, generation, Living Scenes, search, recommendations), and champion–challenger promotion requires a passing eval against a gold set — the same discipline encoded in evaluation-manifest.ts above. For any frontend change, CLAUDE.md mandates visual and functional verification through Claude in Chrome (the browser-automation MCP) — confirming rendering, interactions, layout, CSS, accessibility, and end-to-end user flows, not just that a unit test passed.

Related#

• Content, Localization, Documentation, and Launch Readiness — the localization catalog and the launch/exit gates this page feeds
• Lilith — Contemplative Policy Substrate — where the manuscript-palette domain register is applied
• Trust, Safety, and Privacy — the trust signals the disclosure/evidence behavior tokens encode
• Security, Privacy, and Compliance — the security-readiness manifest companion
• Foundations — shared platform libraries
• V1/features.md § Design System and Experience Quality
• V1/features.md § Testing, QA, Accessibility, Performance, and Security
• Backlog: §29, §30, §31 in V1/TODOS.md