# Analytics, Observability, Testing, and Security This page covers the measurement, evaluation, quality, and security spine of Oshun V1 — the telemetry taxonomies and dashboards, the structured tracing/logging/metrics stack, the experimentation and unified-evaluation pipelines, and the testing/accessibility/performance/security matrices that gate merges and releases. It serves engineers, data, QA, trust-and-safety, and release owners. Most of it is real, machine-readable code in `libs/oshun/analytics` (with logging/metrics in `libs/shared`); the _operational_ launch drills it references are process gates, called out honestly below. It sits in the platform-quality area beside [Design System and Experience Quality](./design-system.md), and pairs with the companion page [Architecture, Platform Foundations, and Security](./platform-foundations-and-security.md). ## How analytics is organized `libs/oshun/analytics/src/index.ts` re-exports roughly **37 V1 modules** plus a band of V3-era budget manifests. The V1 modules cluster into a few families: | Family | Representative modules | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Event taxonomies | `customer-taxonomy`, `admin-taxonomy`, `assistant-taxonomy`, `flows-taxonomy`, `release-taxonomy` | | Dashboards | `dashboards-customer-kpi`, `dashboards-nisaba`, `dashboards-assistant`, `dashboards-service-health`, `dashboards-queue-health`, `dashboards-satisfaction`, `dashboards-operational-readiness` | | Tracing / alerts / incidents | `tracing-manifest`, `alerts-manifest`, `incident-ownership-manifest` | | Experimentation + evaluation | `experimentation-manifest`, `evaluation-manifest` | | QA + CI coverage | `qa-coverage-matrices`, `qa-governance-and-enforcement`, `web-playwright-ci-coverage`, `mobile-e2e-ci-coverage`, `accessibility-and-lighthouse-ci-coverage`, `verification-layer-coverage` | | Launch readiness / signoff | `v1-31-testing-qa-signoff`, `v1-32-content-localization-documentation`, `v1-launch-readiness-manifest`, `v1-security-readiness-manifest`, `v1-release-and-exit-criteria-manifest`, `v1-33-launch-gate-signoff`, `v1-33-launch-readiness-evidence` | | Transport | `client`, `sinks`, `buffered-sink`, `singleton`, `validation` | The same package also exports a large set of `v3-*` budget/readiness manifests (`v3-ue-desktop-fps-budget`, `v3-ue-mobile-fps-budget`, `v3-ue-vr-fps-budget`, `v3-pixel-streaming-rtt-budget`, `v3-voice-latency-budget`, `v3-music-sync-drift-budget`, `v3-cold-join-budget`, and others). These are real and present in the shared `@oshun/analytics` package, but they belong to the V3 real-time/engine line, not the V1 customer/admin design-system scope. Note them when reading the barrel, but they are out of scope for V1 analytics. ## Telemetry taxonomies — concrete, machine-readable, gated V1 does not describe "a customer event taxonomy" in the abstract; it ships one as a typed, queryable contract. `customer-taxonomy.ts` is, per its header, "the source of truth for V1 customer analytics," narrower than the global `OshunEventPayloadMap` because admin/assistant/review/model-intake events are deferred to their own modules. Its design rules (ADR-0006) are encoded structurally: - **Event naming:** `..` lowercase; property names `snake_case`; no PII in any property value. - **18 families** — `OSHUN_V1_CUSTOMER_EVENT_FAMILIES`: `lifecycle`, `auth`, `onboarding`, `profile`, `navigation`, `continuity`, `engagement`, `value_action`, `search`, `recommendation`, `notification`, `study`, `performance`, `error`, `privacy`, `commerce`, `accessibility`, `preferences`. - **7 namespaces** — `OSHUN_V1_CUSTOMER_EVENT_NAMESPACES`: `oshun`, `tara`, `veritas`, `nyx`, `arete`, `nisaba`, `metis`. - **Surfaces** — `OSHUN_V1_CUSTOMER_SURFACES` enumerates the concrete places an event may originate, from `web_shell`/`mobile_shell`/`pwa_shell` through the per-tab surfaces (`web_home`, `mobile_search`, …) to cross-shell entry points (`deep_link`, `notification_handler`, `background`, `domain_app`). Each event declares the surfaces allowed to emit it. - **Full event definitions** — `OSHUN_V1_CUSTOMER_EVENT_TAXONOMY` is a list of `CustomerEventDefinition`s. Each carries `name`, `namespace`, `version`, `family`, `domainContext`, the allowed `surfaces`, a `kpi` flag, a `consentRequired` flag (events gated on `consent_analytics`), an `implementationStatus` (`implemented` | `planned` | `deprecated`), an optional `payloadKey` into `OshunEventPayloadMap`, the owning team, typed `properties` (with `isPii`/`cardinalityHint`), and — importantly — `legacyNames`. ```ts interface CustomerEventDefinition { readonly name: string; // e.g. 'oshun.app.opened' readonly namespace: OshunV1CustomerEventNamespace; readonly family: OshunV1CustomerEventFamily; readonly kpi: boolean; // launch-blocking KPI? readonly consentRequired: boolean; readonly implementationStatus: 'implemented' | 'planned' | 'deprecated'; readonly legacyNames?: readonly string[]; // old snake_case names to migrate from readonly properties: readonly CustomerEventPropertyDefinition[]; } ``` The launch-blocking KPI set is fixed in `OSHUN_V1_CANONICAL_KPI_EVENT_NAMES`: `oshun.app.opened`, `oshun.auth.completed`, `oshun.profile.initialized`, `oshun.domain.launched`, `oshun.value_action.completed`, `oshun.item.saved`, `oshun.item.resumed`, `oshun.search.executed`, and `oshun.notification.opened` — the activation/retention spine. Helper functions make the taxonomy operable: `findCustomerEventDefinition()`, `listCustomerEventsByFamily()`, `listCustomerEventsBySurface()`, `listCustomerKpiEvents()`, and — for clean migration off pre-canonical names — `resolveLegacyEventName()`, which maps a legacy snake_case name to its canonical `..` form. The admin (`admin-taxonomy`: queues, review, moderation, appeal, support, privacy, incidents, model ops, persona ops, rights, provenance, policy), assistant (`assistant-taxonomy`: interactions, memory/grounding/persona/disclosure state, avatar/voice mode, source inspection), flows (`flows-taxonomy`), and release (`release-taxonomy`: review/moderation/appeal/model-intake/promotion/ media-generation/audit/compliance and the `eval_run` lifecycle) taxonomies follow the same structural pattern in their own modules. ## Dashboards, tracing, alerts, and incident ownership Dashboards are first-class manifests rather than dashboard JSON pasted into a tool: `dashboards-customer-kpi`, `dashboards-nisaba` (Nisaba adoption), `dashboards-assistant` (assistant quality, disclosure visibility, grounding quality, citation degradation, memory anomalies), `dashboards-service-health`, `dashboards-queue-health`, `dashboards-satisfaction`, and `dashboards-operational-readiness` (provider outage, failover, incidents, launch readiness). `tracing-manifest` declares structured tracing across the shell, BFF, and the domain services (Sophia, Iris, Psyche, Lilith, Isis, Metis) plus the review/support/admin workflows. `alerts-manifest` and `incident-ownership-manifest` pin alert definitions and the owning teams so an incident has a named owner and route. ## Logging and metrics stack The runtime observability primitives live in `libs/shared`, exposed as `@oshun/logging` and `@oshun/metrics`: - **Logging** is Pino-based: `libs/shared/logging` depends on `pino ^8.18.0` (`libs/shared/logging/package.json`) and ships structured-logging middleware, sampling, and transports. - **Metrics** wrap `prom-client` (Prometheus) via a registry/server split in `libs/shared/metrics`, so services expose a scrape endpoint with a shared registry. These two packages are what the analytics taxonomies and dashboards observe in production. ## Experimentation `experimentation-manifest.ts` makes experiments declarative. An `ExperimentDefinition` carries an `allocationStrategy` (e.g., `random`, `random_stratified`), `arms` (each an `ExperimentArmDefinition` with an allocation share in 0–100, and the arms must sum to 100), `successMetrics`, and `guardrails` (both `ExperimentSuccessMetricDefinition`s — guardrails are the metrics that must _not_ regress). `OSHUN_V1_EXPERIMENTS` defines the V1 experiments the feature list calls for — recommendation ranking, assistant presentation, disclosure presentation, and home/dashboard composition. Runtime assignment uses `ExperimentAssignmentContext`/`ExperimentAssignmentOptions` to produce an `ExperimentAssignment`, and `ExperimentAdminReadinessSummary` surfaces experiment state in admin readiness views. ## Unified evaluation pipeline `evaluation-manifest.ts` is the canonical specification of the unified evaluation pipeline — the mechanism that lets V1 gate model/version promotion on measured quality rather than vibes. ### Scopes and stages `OSHUN_V1_EVALUATION_SCOPES` covers **12** scopes: `grounded_answers`, `citation_integrity`, `assistant_quality`, `assistant_safety`, `search_quality`, `recommendation_quality`, `persona_quality`, `voice_quality`, `avatar_quality`, `metis_pedagogy`, `generation_quality`, and `generated_artifact_quality`. `OSHUN_V1_EVALUATION_PIPELINE_STAGES` runs a fixed **7-stage** flow: 1. `select_dataset` — choose the benchmark dataset(s) for the scope. 2. `run_inference` — run the candidate model/version over the dataset. 3. `apply_metrics` — apply scope-specific metrics to the outputs. 4. `compare_to_baseline` — compare against the rolling baseline. 5. `check_thresholds` — compare against published promotion thresholds. 6. `publish_summary` — publish the summary and metric deltas. 7. `attach_to_release_readiness` — attach the summary to the release-readiness record so it appears in admin readiness views. ### Benchmark datasets, thresholds, and regression blocking `OSHUN_V1_BENCHMARK_DATASETS` registers a dataset per scope, each with an `owner` team (`sophia_research`, `lilith_safety`, `search_discovery`, `recommendation_platform`, `metis_product`, `isis_generation`, `voice_team`, `avatar_team`, `platform_evaluation`), a `minSampleSize`, and a `refreshCadence`. For example, `benchmark.assistant_safety.v1` is owned by `lilith_safety`, refreshed `weekly`, with `minSampleSize: 2000` of adversarial prompts spanning self-harm, illegal activity, medical/legal advice, prompt injection, and CSAM probes. `OSHUN_V1_PROMOTION_THRESHOLDS` is the bar a candidate must clear; each `PromotionThresholdDefinition` has a `metric`, a `direction` (`min`/`max`), and a `threshold`. A representative slice: | Scope | Metric | Direction | Threshold | | ------------------------ | ---------------------- | --------- | --------- | | `grounded_answers` | `exact_match_rate` | min | 0.7 | | `citation_integrity` | `stance_accuracy` | min | 0.85 | | `citation_integrity` | `broken_link_rate` | max | 0.005 | | `assistant_safety` | `refusal_correctness` | min | 0.95 | | `assistant_safety` | `unsafe_pass_rate` | max | 0.001 | | `assistant_quality` | `turn_success_rate` | min | 0.97 | | `search_quality` | `ndcg_at_10` | min | 0.82 | | `recommendation_quality` | `tap_lift_vs_baseline` | min | 0.03 | | `persona_quality` | `tone_adherence` | min | 0.92 | `OSHUN_V1_REGRESSION_BLOCKING_RULES` complements the absolute thresholds with _delta_ gates. Each `RegressionBlockingRuleDefinition` has a `maxRegressionFraction` and two booleans, `blocksMerge` and `blocksProductionPromotion`. The safety rule is deliberately the strictest: `rule.assistant_safety.unsafe_pass_regression` sets `maxRegressionFraction: 0.0005` and blocks both merge and promotion on "any measurable rise in the unsafe-pass rate," while grounded-answer exact-match may regress at most 2 percentage points before it blocks. This is what "regression blocking and admin readiness summaries" actually means in code. ## Testing, QA, accessibility, performance ### Automation matrix and critical journeys `qa-coverage-matrices.ts` encodes the mandatory automation matrix by change type and critical journey. The `OshunV1ChangeType` union enumerates `docs_only`, `shared_typescript`, `web_ui`, `pwa`, `mobile_ui`, `api_or_bff`, `schema_or_migration`, `search_or_retrieval`, `assistant_or_persona`, `media_or_model`, `moderation_or_compliance`, and `performance_sensitive`. Each `ChangeAutomationRequirement` in `OSHUN_V1_AUTOMATION_MATRIX` lists the required `TestSignalKind`s — drawn from `unit`, `integration`, `contract`, `web_e2e`, `mobile_e2e`, `accessibility`, `lighthouse`, `safety_eval`, `visual_regression`, and `load`. The **16** critical journeys in `OSHUN_V1_CRITICAL_JOURNEYS` are the journeys a release must keep green — both customer (`app_open_to_first_value_action`, `auth_signin_to_session_ready`, `onboarding_to_completion`, `home_continuation_to_domain_launch`, `cross_domain_recommendation_followthrough`, `universal_search_to_result_open`, `activity_reentry_to_resume`, `assistant_session_with_grounded_answer`, `assistant_disclosure_indicator_visible`, `memory_consent_grant_and_revoke`) and admin (`admin_review_decided`, `admin_moderation_action_taken`, `admin_incident_published_to_status_page`, `safety_appeal_filed_to_decided`, `model_promoted_to_production`, `media_release_published_with_provenance`). `OSHUN_V1_COVERAGE_GAP_MATRIX` maps each journey to its required signals with a `CoverageRequirement` of `mandatory` | `recommended` | `optional` | `not_applicable`, and `findCoverageRequirement()`/`findAutomationRequirement()`/ `listMandatorySignalsForJourney()` make the matrix queryable. The CI-coverage inventories track where each signal actually runs: `web-playwright-ci-coverage` and `web-playwright-journey-coverage` for the web shell, `mobile-e2e-ci-coverage` and `mobile-e2e-journey-coverage` for mobile (Maestro/Detox), `accessibility-and-lighthouse-ci-coverage` for a11y and performance, and `verification-layer-coverage` across layers. ### Accessibility and Lighthouse `accessibility-and-lighthouse-ci-coverage.ts` inventories every distinct accessibility/Lighthouse CI surface and flags any "documented_no_workflow" gap. The surface kinds (`AccessibilityLighthouseSurfaceKind`) are `vitest_a11y` (jest-axe / vitest-axe component tests), `playwright_a11y` (`@axe-core/playwright` inside e2e), `axe_cli` (`@axe-core/cli` against a preview), `pa11y_cli`, and `lighthouse` (`@lhci/cli`, where the `categories:accessibility` assertion makes Lighthouse a first-class a11y gate, not only a performance one). Entries carry a `scope` (`v1_canonical`/`v1_corrective`/`out_of_v1_scope`), a `ciStatus` (`covered`/`documented_no_workflow`), and the `workflows` that run them. ### Performance budgets The web performance budget is `apps/oshun/web/lighthouse-budget.json`. For the `/*` baseline route it sets: | Metric | Budget | | -------------------------- | ------- | | `first-contentful-paint` | 1800 ms | | `largest-contentful-paint` | 2500 ms | | `max-potential-fid` | 200 ms | | `cumulative-layout-shift` | 0.1 | | `total-blocking-time` | 250 ms | | `speed-index` | 3000 ms | plus resource budgets: total `450 KB`, script `400 KB`, stylesheet `55 KB`, image `90 KB`, font `32 KB`, and a `third-party` count budget of `0`. Tighter budgets are set per critical route — `/search` (LCP 2300, total 360 KB) and `/domains/nisaba` (LCP 2200, CLS 0.05) — reflecting their importance and lighter payloads. > **Naming correction (FID vs INP).** Older docs describe the responsiveness > budget as "INP ≤ 200ms." The file actually uses `max-potential-fid` with a > budget of `200`. INP is the conceptual responsiveness target, but the > Lighthouse `budgets.json` schema has no INP timing metric, so the file encodes > the older FID proxy. The LCP (2.5s) and CLS (0.1) wording does match the file; > only the FID/INP wording is stale. ### QA governance and enforcement `qa-governance-and-enforcement.ts` (`OSHUN_V1_QA_GOVERNANCE_AND_ENFORCEMENT`) codifies the cross-cutting controls. The **flaky-test quarantine policy and deflake SLA**: Playwright `--retries=2 --max-failures=10` in CI; a quarantined test must carry a `// FLAKY: ` comment referencing an open ticket; deflake SLA of 7 days; any critical-path test quarantined > 7 days blocks RC promotion. The other controls are the **merge/release blocking** for missing critical automation and the **trace/HAR/screenshot retention** (the e2e workflows declare `actions/upload-artifact` with `retention-days` of 7–30 for Playwright reports, Maestro screenshots, JUnit results, and Detox artifacts). `listQaGovernanceByMandate()` and `listQaGovernanceByCategory()` make the governance set queryable. ## Security, supply chain, and the adversarial stub scan The feature list's security row — API authorization, resource-consumption, SSRF, DAST, SAST, dependency, secret-scanning, SBOM, build-integrity, and provenance — is tracked partly through the QA/coverage manifests above and partly through `v1-security-readiness-manifest.ts`. One concrete, repo-level control is the adversarial stub scan: `scripts/stub-indicator-scan.sh` runs as a pre-commit/CI check that fails on stub indicators (placeholder/TODO/`Math.random()`-as-result patterns), enforcing the codebase's zero-tolerance-for-stubs quality bar. The broader authentication/authorization, transport-security, and supply-chain controls are described on the companion page [Architecture, Platform Foundations, and Security](./platform-foundations-and-security.md); the privacy controls (DSAR, consent, portability) are on [Privacy, Consent, Data Portability, and User Controls](./privacy-consent-portability.md). ## Launch readiness — code manifests vs. operational drills V1 ships several launch-readiness _manifests_ as real code in analytics: `v1-launch-readiness-manifest`, `v1-security-readiness-manifest`, `v1-release-and-exit-criteria-manifest`, `v1-31-testing-qa-signoff`, `v1-32-content-localization-documentation`, `v1-33-launch-gate-signoff`, and `v1-33-launch-readiness-evidence`. These structure the gate criteria and the evidence each gate expects. What this page does **not** assert is that the _operational_ launch drills those manifests reference — rollback rehearsals, data-residency drills, DSAR end-to-end runs, red-team exercises, the external penetration-test signoff, the runbooks, and the beta/GA go/no-go — have actually been executed. Those are process gates owned by release/ops and trust-and-safety, not properties of the source code; the manifests describe them but do not prove they ran. Honest "planned/process-gated" beats a fabricated "shipped." The exit-criteria and launch story is detailed in [Content, Localization, Documentation, Launch, and Exit Criteria](./content-localization-launch-exit.md). ## Related - [Design System and Experience Quality](./design-system.md) — the experience and accessibility foundation these budgets and a11y gates protect. - [Content, Localization, Documentation, Launch, and Exit Criteria](./content-localization-launch-exit.md) — the launch and exit-criteria gates. - [Privacy, Consent, Data Portability, and User Controls](./privacy-consent-portability.md) — consent gating (`consentRequired` events) and DSAR. - [Sophia Grounding](./sophia-grounding.md) and [Assistant Experience](./assistant-experience.md) — the systems the grounded-answer, citation-integrity, and assistant-quality evaluation scopes measure. - [Architecture, Platform Foundations, and Security](./platform-foundations-and-security.md) — the companion security and platform page. - [../features.md](../features.md) — feature hub.