Eve library

OSHUN Studio Metrics and Analytics Instrumentation Journeys and Edge Cases

Spec

Revision 1

On this page

Status: Active
Owner: OSHUN Studio Architecture + OSHUN Web + OSHUN Data/Analytics Engineering
Last Updated: 2026-02-23
Related:

  • docs/adr/ADR-0040-oshun-studio-metrics-and-analytics-instrumentation.md
  • docs/releases/v1/specs/studio-metrics-and-analytics-instrumentation-dependency-and-data-contract-map.md

Purpose#

Define primary journeys and deterministic edge-case behavior for Studio metrics and analytics instrumentation workflows.

This document is the execution artifact for OST-00219.

Personas#

  • Collaboration Lead
  • Program Operator
  • Domain Integrator
  • Security Reviewer
  • Compliance Auditor

Journey Principles#

  1. Every instrumentation flow must emit canonical requested/completed spans.
  2. Metric definitions and KPI compositions must be deterministic and traceable.
  3. Audit and analytics outcomes must remain contract-linked.
  4. Telemetry quality checks must gate release readiness.

1. Primary Journeys#

J1. Publish Metric Definition#

Goal: Define/update canonical Studio metric semantics and thresholds.

Entry: Metrics instrumentation workspace

Steps:

  1. Select metric type, unit, owner, and aggregation window.
  2. Configure warning and critical thresholds.
  3. Publish metric definition.
  4. Persist versioned metric-definition record.
  5. Emit metric-definition update event.

Success State:

  • Metric definition is stored with deterministic semantics and ownership.

J2. Emit and Correlate Flow Spans#

Goal: Instrument actionable Studio flows with requested/completed spans.

Entry: Flow execution in Studio surface

Steps:

  1. Emit flow_requested event with attribution context.
  2. Execute flow state transition.
  3. Emit flow_completed with success/latency/error metadata.
  4. Correlate span to workflow and source domain.
  5. Persist span lineage for KPI aggregation.

Success State:

  • Flow spans are complete, attributable, and KPI-consumable.

Goal: Preserve deterministic linkage between policy outcomes and analytics.

Entry: Audit event generated by governance flow

Steps:

  1. Capture audit action and outcome.
  2. Map audit record to canonical analytics event contract.
  3. Persist linkage metadata (auditEventId, analyticsEventName).
  4. Validate linkage integrity and null-rate thresholds.
  5. Emit linkage completion event.

Success State:

  • Every critical audit outcome is analytics-linked and queryable.

J4. Compose and Validate KPI Signals#

Goal: Build KPI aggregates for reliability, latency, and policy integrity.

Entry: Metric and span streams available

Steps:

  1. Select KPI composition model (rate, ratio, p95, sum).
  2. Bind numerator/denominator metric sources.
  3. Calculate KPI values for configured windows.
  4. Compare values against warning/critical thresholds.
  5. Emit KPI evaluation result and threshold status event.

Success State:

  • KPI outputs are deterministic, threshold-aware, and release-gate ready.

J5. Run Telemetry Quality and Release Gate#

Goal: Validate telemetry completeness before production promotion.

Entry: Release candidate instrumentation bundle

Steps:

  1. Execute schema validation on instrumentation payloads.
  2. Evaluate completeness, null-dimension, and duplication rates.
  3. Verify flow-span and audit-linkage coverage.
  4. Generate release-gate verdict with evidence references.
  5. Emit telemetry-quality gate decision event.

Success State:

  • Release gate passes only when telemetry quality thresholds are met.

2. Critical Edge Cases#

E1. Metric Definition Missing Owner#

  • Trigger: metric definition published without owner.
  • Required behavior: reject publish and require owner attribution.

E2. Metric Threshold Inversion#

  • Trigger: critical threshold set below warning threshold.
  • Required behavior: block publish and return deterministic validation error.

E3. Flow Requested Without Completion#

  • Trigger: requested span emitted but completed span missing.
  • Required behavior: flag orphan span and trigger instrumentation incident.

E4. Duplicate Flow Completion Emission#

  • Trigger: same span completes multiple times.
  • Required behavior: enforce idempotent completion and retain one canonical record.

E5. Audit Outcome Without Analytics Linkage#

  • Trigger: audit event persists without linkage record.
  • Required behavior: block release gate and emit quality-degradation event.

E6. KPI Composition Missing Denominator#

  • Trigger: ratio KPI configured without denominator metric.
  • Required behavior: reject KPI evaluation and require complete composition spec.

E7. High Null-Dimension Rate#

  • Trigger: required attribution fields exceed null-rate threshold.
  • Required behavior: mark telemetry quality failed and prevent production release.

E8. Domain Mapping Drift#

  • Trigger: source domain not in canonical registry during ingestion.
  • Required behavior: quarantine event and emit schema-mismatch alert.

E9. PII Field Detected in Payload#

  • Trigger: payload includes disallowed PII property.
  • Required behavior: reject ingestion and emit privacy-policy violation event.

E10. KPI Threshold Alert Not Routed#

  • Trigger: critical KPI breach without alert routing.
  • Required behavior: trigger fallback alert routing and incident marker.

3. Journey-to-Surface Mapping#

Journey Primary Surface Supporting Surfaces
J1 Publish Metric Definition Metrics/analytics instrumentation Analytics contract services
J2 Emit and Correlate Flow Spans Metrics/analytics instrumentation Studio flow surfaces + BFF
J3 Link Audit Outcomes Metrics/analytics instrumentation Audit/compliance + analytics linkage
J4 Compose and Validate KPI Signals Metrics/analytics instrumentation KPI aggregation and threshold engine
J5 Run Telemetry Quality Release Gate Metrics/analytics instrumentation Release governance + quality checks

4. Instrumentation Requirements#

Required events:

  • metrics_analytics_instrumentation.journey.started
  • metrics_analytics_instrumentation.journey.step.completed
  • metrics_analytics_instrumentation.journey.blocked
  • metrics_analytics_instrumentation.journey.recovered
  • metrics_analytics_instrumentation.journey.completed

Required dimensions:

  • metricId, metricType, spanId, auditEventId, kpiId, sourceDomain
  • actorId, actorRole, workspaceId, releaseChannel, qualityScore
  • journeyId, stepId, outcome, latencyMs, errorCode

5. Acceptance Criteria (OST-00219)#

OST-00219 is complete only when:

  1. At least five metrics/instrumentation journeys are documented.
  2. At least ten deterministic edge cases are documented.
  3. Journey-to-surface mapping is explicit.
  4. Instrumentation requirements and dimensions are documented.