# Observability, Design System, Testing, and Performance

This page covers the platform-quality spine that sits underneath every Oshun V1
surface: how the system is observed and evaluated, how its visual and
interaction language is encoded as code, how it is tested, and how its
performance is budgeted and enforced. These concerns are cross-cutting — they
serve every customer-facing domain (Tara, Veritas, Nyx, Arete, Nisaba, Metis),
the admin/Studio surfaces, and the substrates (Sophia, Iris, Psyche, Lilith,
Isis, Aje) — and they are where "is V1 ready to ship?" is actually answered. For
the most part this is implemented, tested code, not a spec: the i18n catalog,
analytics taxonomies and manifests, design tokens, the UI component library, and
the design-language audit modules are all real, sizeable, exercised libraries.
What remains genuinely operational — the launch drills, pen-test signoff, and
go/no-go gates — is described candidly as process in
[Content, Localization, Documentation, and Launch Readiness](./content-localization-and-launch.md).

Product scope for the design system lives in
[`V1/features.md` § Design System and Experience Quality](../features.md#design-system-and-experience-quality);
testing/QA/performance scope in
[`V1/features.md` § Testing, QA, Accessibility, Performance, and Security](../features.md#testing-qa-accessibility-performance-and-security).
Backlog: §29 (design system), §31 (testing/QA), plus the observability and
evaluation items in §30. The hub is [../ARCHITECTURE.md](../ARCHITECTURE.md).

---

## Observability, Evaluation, and Release Gates

Observability in V1 is the standard three-pillar setup — traces, logs, metrics —
wired through shared libraries so that every service emits the same shapes, and
backed by named, machine-readable evaluation and release manifests so that
"ready to promote" is a checkable fact rather than a judgment call.

### Tracing, logging, metrics

- **Structured tracing** spans the shell, the BFF, and the substrates — Sophia,
  Iris, Psyche, Lilith, Isis — plus the domain services. Spans export over OTLP
  via `OTEL_EXPORTER_OTLP_ENDPOINT`, and the trace UI is Jaeger.
- **Logging** flows through `@oshun/logging` (package at
  `libs/shared/logging/`), which is a **Pino** logger (`pino ^8.18.0`) with the
  surrounding middleware, sampling, and transport plumbing — its source carries
  `logger.ts`, `events.ts`, and per-module tests (`logger.spec.ts`).
- **Metrics** flow through `@oshun/metrics` (`libs/shared/metrics/`), which
  wraps **prom-client** (`prom-client` via the pnpm catalog) behind a shared
  registry — `registry.ts` plus a metrics `server.ts` expose the Prometheus
  scrape surface, and `helpers.ts` standardizes counter/histogram construction.
  Dashboards land in Grafana.

The point of routing everything through this shared plumbing is uniformity: a
trace from Sophia's retrieval path and a trace from the assistant's tool loop
use the same field names, so a single Grafana/Jaeger query works across
substrates, and an SLO breach alert fires off one canonical metric name rather
than a per-team dialect.

### Evaluation pipeline — `evaluation-manifest.ts`

The evaluation discipline is encoded concretely in
`libs/oshun/analytics/src/evaluation-manifest.ts`. It is not prose: it is a set
of exported constants the CI/CD release gates consume directly.

`OSHUN_V1_EVALUATION_SCOPES` enumerates the twelve things V1 evaluates, so no
launching surface can quietly ship without a scope:

| #   | Scope                        | What it gates                     |
| --- | ---------------------------- | --------------------------------- |
| 1   | `grounded_answers`           | Answer grounding quality          |
| 2   | `citation_integrity`         | Citations actually support claims |
| 3   | `assistant_quality`          | Assistant response quality        |
| 4   | `assistant_safety`           | Assistant safety behavior         |
| 5   | `search_quality`             | Search relevance/ranking          |
| 6   | `recommendation_quality`     | Recommendation relevance          |
| 7   | `persona_quality`            | Persona fidelity                  |
| 8   | `voice_quality`              | Voice-pack fidelity               |
| 9   | `avatar_quality`             | Avatar-pack fidelity              |
| 10  | `metis_pedagogy`             | Metis lesson/pedagogy quality     |
| 11  | `generation_quality`         | Generation request quality        |
| 12  | `generated_artifact_quality` | Output-artifact quality           |

`OSHUN_V1_EVALUATION_PIPELINE_STAGES` defines the seven ordered stages every
eval run walks through:

```
select_dataset → run_inference → apply_metrics → compare_to_baseline
  → check_thresholds → publish_summary → attach_to_release_readiness
```

The final stage — `attach_to_release_readiness` — is the load-bearing one: it
binds an eval result to the release record so that a green eval is _evidence on
the release_, not a transient log line. The companion constants
`OSHUN_V1_BENCHMARK_DATASETS` (the dataset registry, queryable by scope and by
id via the exported helpers), `OSHUN_V1_PROMOTION_THRESHOLDS` (per-scope bars a
candidate must clear), and `OSHUN_V1_REGRESSION_BLOCKING_RULES` (the rules that
fail a build closed when a metric regresses) complete the loop. The file header
ties these to the backlog ids `V1-OBS-020`…`V1-OBS-023`.

### Experimentation and champion–challenger

Rollouts for the assistant, generation, and search use **champion–challenger**
promotion: a challenger only takes over once it clears a passing eval against a
gold set. The experiment shape is the exported `ExperimentDefinition` in
`libs/oshun/analytics/src/experimentation-manifest.ts`, which carries `arms`
(each with an allocation share that the arms must sum to 100), `successMetrics`,
`guardrails`, and an `allocationStrategy` (e.g. `random_stratified`). The live
catalog `OSHUN_V1_EXPERIMENTS` instantiates these definitions, so the
experimentation surface is data, not an ad-hoc dashboard. Living Scenes carry
their own eval sets and release gates, described in
[`V1/features.md` § Eval Sets, Release Gates, and Rollback](../features.md).

### The broader analytics package

`libs/oshun/analytics/src/index.ts` re-exports roughly three dozen V1 modules.
Beyond evaluation and experimentation, the V1-scoped set includes the customer
event taxonomy, `admin-taxonomy`, `assistant-taxonomy`, `flows-taxonomy`,
`release-taxonomy`, a `tracing-manifest`, an `alerts-manifest`, an
`incident-ownership-manifest`, the QA coverage matrices (`qa-coverage-matrices`,
`qa-governance-and-enforcement`), the CI-coverage manifests for web Playwright
(`web-playwright-ci-coverage`), mobile e2e (`mobile-e2e-ci-coverage`), and
accessibility/Lighthouse (`accessibility-and-lighthouse-ci-coverage`), plus the
signoff/readiness manifests `v1-31-testing-qa-signoff`,
`v1-launch-readiness-manifest`, `v1-security-readiness-manifest`,
`v1-release-and-exit-criteria-manifest`, `v1-33-launch-gate-signoff`, and
`v1-33-launch-readiness-evidence`. Note that the same package also still carries
a large block of **V3-era budget manifests** (`v3-ue-desktop-fps-budget`,
`v3-ue-mobile-fps-budget`, `v3-ue-vr-fps-budget`,
`v3-pixel-streaming-rtt-budget`, `v3-voice-latency-budget`,
`v3-music-sync-drift-budget`, `v3-cold-join-budget`, and more) — these are
present in the tree but are out of V1 design-system scope; do not read them as
V1 surfaces.

#### Customer event taxonomy — `customer-taxonomy.ts`

The customer analytics contract is concrete and named in
`libs/oshun/analytics/src/customer-taxonomy.ts`.
`OSHUN_V1_CUSTOMER_EVENT_FAMILIES` is the closed set of eighteen event families:

`lifecycle`, `auth`, `onboarding`, `profile`, `navigation`, `continuity`,
`engagement`, `value_action`, `search`, `recommendation`, `notification`,
`study`, `performance`, `error`, `privacy`, `commerce`, `accessibility`,
`preferences`.

`OSHUN_V1_CUSTOMER_EVENT_NAMESPACES` is the seven-namespace partition that
scopes events to the platform shell and each consuming domain: `oshun`, `tara`,
`veritas`, `nyx`, `arete`, `nisaba`, `metis`. The full event definitions live in
`OSHUN_V1_CUSTOMER_EVENT_TAXONOMY` (each carrying `legacyNames` so older event
names migrate cleanly), the canonical KPI subset is
`OSHUN_V1_CANONICAL_KPI_EVENT_NAMES`, and `resolveLegacyEventName()` resolves an
old name to its canonical form. This is the machinery behind the "activation,
retention, value-action" KPIs — but spelled out as a typed, test-locked taxonomy
rather than a loose convention.

---

## Design System and Experience Quality

The design system is encoded as code, with a single source of truth for tokens,
a real React component library, and a separate suite of _design-language audit_
modules that check the system against itself.

### Design tokens — `libs/oshun/design-tokens/`

The canonical token source is `libs/oshun/design-tokens/src/tokens.ts`. It is
worth being precise about what the canonical surface actually _is_, because the
older architecture text had it inverted.

The **canonical** V1 color system is a **cool, dark ink/fog + aqua/amber**
palette, not "cream-paper + terracotta". `oshunTokenSchema.color` groups the
scales as:

| Group     | Scales              | Anchor values                                     |
| --------- | ------------------- | ------------------------------------------------- |
| `neutral` | `ink`, `fog`        | `ink[950] = #040B16`, `ink[900] = #0B1728`        |
| `brand`   | `aqua`, `amber`     | `aqua[500] = #20C9D8`, `amber[600] = #DD8618`     |
| `status`  | `success`, `danger` | `success[500] = #2AB673`, `danger[500] = #E24D4D` |

The **default theme is dark**: `defaultOshunTheme = oshunThemes.dark`. The
`oshunThemes` map is keyed by `OshunThemeName` — `'light'`, `'dark'`,
`'highContrastLight'`, `'highContrastDark'` — and `getSemanticTokens()` /
`createCssVariables()` default to that dark theme. The foundation bundle is
versioned: `OSHUN_V1_FOUNDATION_TOKEN_BUNDLE_ID = 'oshun-v1-foundation'` at
`OSHUN_V1_FOUNDATION_TOKEN_BUNDLE_VERSION = '1.0.0'`.

The **cream-paper-and-ink + terracotta** language people remember is the
**Lilith register** (the "manuscript palette"), and it lives specifically inside
the `color.domain` block of `oshunTokenSchema` (around line 1103), where each
customer domain is mapped to a manuscript hue:

| Domain    | Hue       | Note                    |
| --------- | --------- | ----------------------- |
| `tara`    | `#5A7A3F` | Lilith green            |
| `veritas` | `#9A3E1C` | terracotta / burnt rust |
| `nyx`     | `#3B3325` | ink-2                   |
| `arete`   | `#3F5E2B` | deep green              |
| `nisaba`  | `#C89657` | honey / tan             |
| `metis`   | `#6B5A44` | warm brown              |

So the relationship is the reverse of the old claim: the ink/fog + aqua/amber
dark system is the canonical surface, and the manuscript palette is the domain
accent register applied at the persona-policy boundary (see
[Lilith — Contemplative Policy Substrate](./substrate-lilith.md)). The in-code
comment is explicit that these domain accents are "mapped into the Lilith
register … so the sidebar reads as the same cream-paper-and-ink language as the
rest of the surface" — i.e. the manuscript feel is applied _to_ the canonical
system, not the other way around.

Tokens are not only colors. The same file exports the spacing scale
(`oshunSpacingScale`), `oshunFontWeights`, the `oshunTypographyRamp`, the motion
system (`oshunMotionDurationScale`, `oshunMotionTokens`), elevation
(`oshunElevationTokens`), and the domain-accent behavior rules
(`oshunDomainAccents`, `oshunDomainAccentBehavior` — which encodes that
`success`/`warning`/`danger`/`info` always outrank domain identity for critical
meaning). A further family of _behavioral_ token sets encodes how trust-bearing
UI must behave: `oshunGroundedEvidenceUiBehavior`, `oshunDisclosureUiBehavior`,
`oshunAssistantPersonaSwitchingUiBehavior`, and `oshunTrustSignalUiBehavior`.
These are the policy-as-tokens that keep, for example, an AI-disclosure chip
from ever being styled like a celebratory success badge. Mobile derives the same
tokens through `@oshun/design-tokens` rather than maintaining a parallel
palette.

### Component library — `libs/oshun/ui/`

`libs/oshun/ui/src/index.ts` re-exports three subpackages — `components/`,
`motion/`, and `theme/`. The component surface is concrete: the `components/`
directory ships roughly twenty-one React components — `ActionSheet`, `Badge`,
`Banner`, `BottomNav`, `BottomSheet`, `Box`, `Button`, `Card`, `Chip`,
`CommandPalette`, `DomainSwitcher`, `EmptyState`, `ErrorState`, `IconButton`,
`ListRow`, `Modal`, `Stack`, `StatTile`, `Text`, `Toast`, and `TopBar` — grouped
(data-display, feedback, navigation, overlays, primitives), each with their own
`.stories.tsx` and `.test.tsx`, and an `accessibility-snapshots.test.tsx` that
locks the a11y behavior of the whole set. These are the shared components
consumed by every Oshun web/admin surface, so a `DomainSwitcher` or
`CommandPalette` looks and behaves identically across domains.

### Information architecture

Shell ordering is Tara-led — the registry's `OSHUN_SHELL_PRIMARY_DOMAIN` places
Tara at the center — and domain switching is a single gesture: one tap on mobile
via `DomainSwitcher`/`BottomNav`, one keystroke (`Cmd-K`) on web via
`CommandPalette`. The responsive grid pairs a desktop multi-panel workspace
(`apps/oshun/web/src/app/(workspace)/`) with a one-handed mobile IA; both pull
their breakpoints from the design tokens rather than ad-hoc media queries.

### Accessibility as a baseline

WCAG 2.1 AA is the V1 minimum, not an aspiration: contrast, reduced-motion
variants, large-text variants, screen-reader controls, and keyboard-only flows.
The token system carries `highContrastLight`/`highContrastDark` themes for the
contrast case, and the design-token motion set carries the reduced-motion path.
Tara audio and Living Scenes carry transcripts, captions, and
alternative-modality variants; Living Scenes ship reduced-motion renders per
[`V1/features.md` § Reduced-Motion and Accessibility Renders](../features.md).

### Design-language audit modules — `libs/oshun/design-language/`

A frequently-missed part of the system is that the design _language_ is audited
by code. `libs/oshun/design-language/src/` ships a suite of modules, each with a
matching `.test.ts`:

| Module                  | Role                                               |
| ----------------------- | -------------------------------------------------- |
| `component-catalog.ts`  | The required-component contract and coverage check |
| `scorecards.ts`         | Per-surface design scorecards                      |
| `ergonomics.ts`         | Interaction-ergonomics rules                       |
| `responsive-system.ts`  | Breakpoint/responsive rules                        |
| `surface-language.ts`   | Surface/visual-language rules                      |
| `voice-and-tone.ts`     | Copy voice/tone rules                              |
| `ai-disclosure-copy.ts` | Canonical AI-disclosure copy                       |
| `failure-copy.ts`       | Canonical failure/error copy                       |
| `visual-qa.ts`          | Visual-QA checks                                   |
| `completion-audit.ts`   | Roll-up completion audit                           |

The catalog is the strongest of these. `component-catalog.ts` exports
`OSHUN_COMPONENT_CATALOG_VERSION = 'component-catalog-v1'`,
`OSHUN_COMPONENT_CATALOG` (every cataloged component, each carrying its
`requirement`, `group`, `canonicalExport`, `implementationPath`, `testPath`,
`customerShell`/`adminShell` flags, and `requiredA11yChecks`), and
`OSHUN_REQUIRED_COMPONENT_REQUIREMENTS` — the 22 requirements that _must_ be
covered: `domain-switcher`, `continuity-card`, `evidence-card`,
`evidence-sidebar`, `claim-item`, `source-item`, `passage-item`, `notebook`,
`collection`, `assistant-disclosure`, `memory-state`, `provenance-badge`,
`admin-queue`, `admin-filter`, `incident-severity`, `sla-timer`,
`review-decision`, `policy-diff`, `avatar`, `voice-pack`, `rights-summary`, and
`license-summary`. The exported coverage function checks the catalog against
this required set and reports `totalRequirements`/`coveredRequirements`, so a
missing trust-bearing component (say, `provenance-badge`) fails the audit rather
than slipping out silently.

---

## Testing, Accessibility, and Performance

### Unit, integration, and e2e

Unit and integration tests use **Vitest** — and per `CLAUDE.md`, the way to run
a single suite while Nx is degraded by worktrees is
`npx vitest run path/to/test.ts` from the library directory. E2E is
**Playwright** across customer web, admin web, and Studio subroutes;
keyboard-only journeys, visual regression, and resilience coverage are
launch-gating, and the expected coverage is itself recorded as data in the
analytics package's `web-playwright-ci-coverage`, `mobile-e2e-ci-coverage`, and
the journey-coverage manifests. Accessibility is automated with
`@axe-core/playwright` and backed by manual audits against WCAG 2.1 AA.

> Honest status: the coverage and budget _manifests_ exist and are test-locked,
> but this audit verified the files, not a live green Playwright/Lighthouse/axe
> run. Whether those checks currently pass on `main` is an operational fact, not
> one provable from the source tree alone.

### Performance budgets — `apps/oshun/web/lighthouse-budget.json`

Customer-web performance is enforced by `apps/oshun/web/lighthouse-budget.json`,
a per-route Lighthouse budgets file. The root `/*` budget is:

| Metric (file key)          | Budget  | Conceptual target             |
| -------------------------- | ------- | ----------------------------- |
| `first-contentful-paint`   | 1800 ms | FCP                           |
| `largest-contentful-paint` | 2500 ms | **LCP ≤ 2.5 s**               |
| `max-potential-fid`        | 200 ms  | **input latency (FID proxy)** |
| `cumulative-layout-shift`  | 0.1     | **CLS ≤ 0.1**                 |
| `total-blocking-time`      | 250 ms  | TBT                           |
| `speed-index`              | 3000 ms | Speed Index                   |

Resource budgets cap the page at **450 KB total / 400 KB script** (with 55 KB
stylesheet, 90 KB image, 32 KB font, and **zero third-party** resources). Hot
routes tighten the bars further — `/search` drops to 360 KB total and 2300 ms
LCP, and `/domains/nisaba` to 355 KB total, 2200 ms LCP, and 0.05 CLS.

One correction to the older docs: the conceptual interaction target is **INP**,
but the file does _not_ encode an INP timing metric — Lighthouse `budgets.json`
has no INP key. The actual enforced key is **`max-potential-fid` at 200 ms**,
the older FID proxy. So "INP ≤ 200 ms" is the intent; "`max-potential-fid` ≤
200" is what the file checks. LCP (2.5 s) and CLS (0.1) match the conceptual
targets directly. Mobile budgets follow Expo's perf baseline plus battery,
memory, and caching budgets per the device-matrix run.

### Tenant isolation and the stub scan

Tenant isolation is launch-gating: `tests/security/tenant-isolation/` must pass,
and any cross-tenant cache, feature-flag, or experiment leakage is treated as a
P0. Separately, the repo's **adversarial stub scan** lives at
`scripts/stub-indicator-scan.sh` — it runs at pre-commit on staged files and in
CI on the whole PR diff, filtering `.ts`, `.tsx`, `.rs`, and `.py`, and exits
non-zero on actionable stub markers (`--mode=stubs`), with `--mode=random` for
`Math.random()` in implementation source and `--mode=all` for both. Its header
ties it to CI jobs `stub-indicator-scan` and `silent-random-scan`, a nightly
`adversarial-grep-trend`, and the pre-commit hook.

### Eval sets and frontend verification

Every V1-launching surface carries a per-surface eval harness (assistant,
generation, Living Scenes, search, recommendations), and champion–challenger
promotion requires a passing eval against a gold set — the same discipline
encoded in `evaluation-manifest.ts` above. For any frontend change, `CLAUDE.md`
mandates visual _and_ functional verification through Claude in Chrome (the
browser-automation MCP) — confirming rendering, interactions, layout, CSS,
accessibility, and end-to-end user flows, not just that a unit test passed.

---

## Related

- [Content, Localization, Documentation, and Launch Readiness](./content-localization-and-launch.md)
  — the localization catalog and the launch/exit gates this page feeds
- [Lilith — Contemplative Policy Substrate](./substrate-lilith.md) — where the
  manuscript-palette domain register is applied
- [Trust, Safety, and Privacy](./trust-safety-and-privacy.md) — the trust
  signals the disclosure/evidence behavior tokens encode
- [Security, Privacy, and Compliance](./security-privacy-compliance.md) — the
  security-readiness manifest companion
- [Foundations](./foundations.md) — shared platform libraries
- [`V1/features.md` § Design System and Experience Quality](../features.md#design-system-and-experience-quality)
- [`V1/features.md` § Testing, QA, Accessibility, Performance, and Security](../features.md#testing-qa-accessibility-performance-and-security)
- Backlog: §29, §30, §31 in [`V1/TODOS.md`](../TODOS.md)