# OBS-001 evidence pack — cross-version GenAI tracing

> **Status: repository-complete / closed.** The repository now has one strict,
> metadata-only OpenTelemetry GenAI vocabulary joined to the RUN-001 common run
> spine; exact V1, V6, V8, and V9 adapter/product mapping; live V6 cognition
> workflow, agent, provider, and tool-engine spans; real OTLP/HTTP trace and
> metric exporters; and deterministic graph diagnostics for every failure class
> named by OBS-001. This closure does not claim a deployed collector, production
> trace sample, or production alert result.

## Work-package charter

- **Task ID:** `OBS-001`
- **Owner:** cross-version runtime/observability maintainers and V6 cognition
  and welfare maintainers; implementation owner: Elikem Agudogo
- **Accountable product:** shared V1–V9 autonomous-content runtime, with V6
  cognition and welfare as the invoked reference integration
- **Priority:** P1
- **Target milestone:** 2026-07-15 SOTA gap-closure program
- **Risk tier:** high release-proof, incident-diagnosis, tenant-isolation, and
  sensitive-data exposure risk
- **Dependencies:** `RUN-001` common run identity; `PRIV-001` metadata-only
  sensitive-data policy; `OPS-003` release/SLO operations; OpenTelemetry API,
  trace, metrics, resources, OTLP/HTTP, and semantic-convention packages; V6
  cognition and welfare runtime boundaries
- **Affected schemas and content slots:** GenAI trace schema version 1, RUN-001
  common-run schema/identity version 1, shared trace and metric exporter APIs,
  V6 cognition trace correlation, and V6 welfare telemetry. No rendered customer
  content slot, prompt body, model output, UI, media, or persisted production
  record changed.
- **Linked task IDs:** `OBS-001`, `RUN-001`, `PRIV-001`, and `OPS-003`
- **Audit finding:** `@oshun/tracing` already depended on the OpenTelemetry SDK,
  but the shared AI helper used legacy `ai.*` attributes, explicit parents were
  dropped by both tracer implementations, lightweight concurrency used one
  process-global active span, production could silently construct a tracer with
  no exporter, and V6 welfare telemetry stopped at an honest in-memory draft.
  The repository also lacked a typed trace graph and release-completeness audit.
- **Repository objective:** make cross-version GenAI operations and Oshun
  lineage queryable through exact common-run identity, export real OTLP from the
  actionable V6/local-only seams, and fail deterministically on incomplete or
  ambiguous trace graphs.
- **Rollout strategy:** additive canonical contracts and shared helpers; V6
  cognition is wired at its governed runtime boundary and V6 welfare gains an
  explicit exporter command. Production tracer initialization is fail-closed
  unless the service identity and OTLP collector are configured.
- **Rollback strategy:** revert source commit
  `f2a69ad905f3704ccc83e80b460dd8ee4ee30dad`.
- **Evidence-pack location:** this file.

## Closure matrix

| Ledger requirement                                                                                                                          | Closure proof                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Adopt OpenTelemetry GenAI conventions for agent, workflow, generation, retrieval, tool, model, provider, token, and conversation operations | `genai-trace.ts` and `@oshun/tracing/genai.ts` pin the repository baseline to OpenTelemetry semantic conventions 1.41.0 and implement `invoke_agent`, `invoke_workflow`, `chat`, `generate_content`, `text_completion`, `embeddings`, `retrieval`, and `execute_tool`. Operation-specific validation requires the relevant agent, workflow, data-source, tool, provider, request/response model, token-usage, and conversation attributes. The vocabulary follows the official [GenAI semantic-conventions repository](https://github.com/open-telemetry/semantic-conventions-genai); the main semantic-conventions page records that GenAI conventions moved there. |
| Extend traces with all named Oshun IDs                                                                                                      | `OSHUN_TRACE_ATTRIBUTE_KEYS`, `GenAiCorrelationIds`, and `GenAiTraceCorrelationSchema` cover slot, artifact, source, canon, gate, rights, policy, intervention, publication, placement, canary, invalidation, runtime-event, provider-event, and engine-event IDs. Inputs are bounded, non-empty, unique, and metadata-only. Exact common-run records automatically contribute aggregate artifact, gate, and intervention IDs.                                                                                                                                                                                                                                       |
| Correlate V1, V6, V8, V9, product runtime, provider, and engine events through the common run spine                                         | One canonical `genAiProductVersionForAdapter` mapping prevents adapter/product substitution for both contracts and runtime helpers. `traceCommonRunOperation` derives adapter, product, and `run_<sha1>` identity from the exact adapter record rather than caller input. The cross-version matrix covers both V1 adapters, V6, V8, and both V9 adapters with product-runtime/provider/engine spans. The live V6 gateway test proves workflow → agent → two provider generations plus a tool-engine effect share one trace and exact common run.                                                                                                                     |
| Export real OTLP telemetry from V6 and every production path that was local/in-process only                                                 | The shared tracer now resolves standard OTLP trace environment variables, uses parent-based ratio sampling and a real OTLP/HTTP exporter, and rejects disabled/missing-collector/missing-service production configuration. `exportOtlpMetrics` performs an awaited OTLP/HTTP metric export and returns only an endpoint hash and bounded receipt. `exportEgbeBehaviorWelfareTelemetryBatch` converts the existing sanitized V6 draft into real metric points without relabeling the draft itself as exported; `pnpm run export:v6:behavior-welfare-telemetry` is the operational entry point. Local collectors received actual OTLP JSON trace and metric documents. |
| Detect missing spans, broken parentage, duplicate effects, orphan artifacts, and untraceable released content                               | `evaluateGenAiTraceCompleteness` parses bounded strict inputs, detects missing required operation/component spans, absent/cross-run/cyclic/duplicate parentage, multiple successful executions of one effect ID, absent or cross-run artifact parents/producers, and released content lacking one successful exact artifact/publication/placement/gate/rights join. The adversarial test produces all five mandated diagnostic kinds in one graph.                                                                                                                                                                                                                   |

## Semantic-convention and privacy boundary

The canonical span contract retains semantic metadata, typed references, low
cardinality error types, and counts. It has no prompt, message, transcript,
source-body, tool-input, tool-output, or model-output field. Runtime helpers
likewise never accept those bodies as attributes. V6 conversation identity and
retrieved-context/memory references are SHA-256-derived typed references; the
live test proves the steward prompt and remembered text do not appear anywhere
in recorded spans.

Operation rules fail before the traced operation runs:

- generation requires `gen_ai.provider.name` and `gen_ai.request.model`;
- agent invocation requires provider, agent ID, and agent name;
- workflow invocation requires workflow name;
- retrieval requires `gen_ai.data_source.id`;
- tool execution requires tool name, with call/effect IDs when available; and
- chat requires a conversation ID.

Token counts must be non-negative integers. Correlation lists reject empty or
duplicate references. A V6 adapter cannot be relabeled as V9 (or any other
product), and the same rule is centralized for every V1/V6/V8/V9 adapter.

## Runtime trace topology

For an opted-in governed V6 cognition run, `AgentRunManager` first creates the
exact V6 common adapter record and then emits:

```text
invoke_workflow v6.cognition                      product_runtime
└── invoke_agent egbe:<tier>                      product_runtime
    ├── generate_content <requested model>        provider
    ├── execute_tool <tool name>                  engine
    └── generate_content <requested model>        provider
```

Every span carries the adapter-derived common run, common cognition step, V6
product, and its component identity. Provider calls add actual response model
and input/output token counts. Tool spans add a deterministic effect ID and
`succeeded`/`failed` outcome without retaining arguments or results. Runtime,
provider, and engine event references make the three seams independently
queryable.

The shared tracer fixes the underlying propagation defects as part of this
closure:

- OTel `SpanContext` input is installed as the actual parent context;
- root spans delete the active parent rather than passing an ignored option;
- OTel and callback records preserve parent, kind, links, attributes, status,
  and updated names;
- an explicit error status is not overwritten by automatic success handling;
- lightweight spans use `AsyncLocalStorage`, so concurrent children retain the
  correct parent and trace; and
- W3C injection observes the currently active lightweight span.

## Real OTLP export and fail-closed configuration

Two independent local-collector integration tests cross the network boundary:

1. the trace exporter sends one OTLP/HTTP JSON document containing a root,
   nested GenAI generation, and explicitly parented child; the exported parent
   and trace IDs are asserted;
2. the metric exporter sends an OTLP/HTTP JSON `resourceMetrics` document and
   returns a receipt containing the endpoint SHA-256, point count, metric names,
   and export time.

V6 welfare adds a third domain-level wire test. It exports all sanitized bond,
arc, departure, chronicle, conduct, and per-tier cost points, proves the body is
an OTLP metric document, and proves the deliberately present raw consent fixture
text is absent. The source batch remains `transport: in-memory-batch-draft`,
`intendedExporter: otlp-http`, and `exported: false`; only a successful awaited
exporter call returns an `exported: true` receipt.

`createTracerConfigFromEnv` rejects production when tracing is disabled, when
the OTLP endpoint is absent, or when service identity is absent. The V6 exporter
command similarly exits non-zero when no metric endpoint is configured; the
negative command was executed and returned `OTLP metrics endpoint is required`.

## Trace-completeness diagnostics

| Adversarial fixture                                                                          | Result                                                      |
| -------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Required operation or component absent                                                       | `missing_span` with run/operation or run/component subject. |
| Parent missing, cross-run, cyclic, duplicated, or adapter/product-substituted                | `broken_parentage`.                                         |
| Same effect ID has more than one successful execution span                                   | `duplicate_effect`.                                         |
| Artifact producer absent/unjoined or artifact parent absent/cross-run                        | `orphan_artifact`.                                          |
| Released artifact lacks one successful exact artifact/publication/placement/gate/rights span | `untraceable_release`.                                      |
| Prompt/source/output body added to the strict canonical span                                 | Schema rejects the extra field.                             |
| Duplicate correlation ID or malformed canonical run ID                                       | Schema/helper rejects before execution.                     |
| Explicit remote parent supplied to lightweight or OTel tracer                                | Child retains exact parent and trace IDs.                   |
| Concurrent lightweight children                                                              | Both retain the active root; no process-global parent race. |
| Tool reports an error without throwing                                                       | Effect outcome is failed and error status remains ERROR.    |

## Automated verification

### Current admitted refresh

The current refresh ran against repository commit
`b45f4cb6ae5439091f8da2a44980210422b9319c` with lockfile SHA-256
`08b055786e9b8a6104d5313601d7eef3205dedcf395066fd9a8264c9859d690f` on host
`app-dev-server`, Ubuntu 24.04.4 LTS, Linux `6.8.0-134-generic` x86_64. The
relevant toolchain was Node.js `v24.15.0`, pnpm `10.25.0`, TypeScript `6.0.3`,
and Vitest `4.1.7`. `@oshun/tracing` was `0.1.0`; installed OpenTelemetry
versions were API `1.9.1`, semantic conventions `1.41.1`, trace
SDK/resources/context `2.7.1`, trace OTLP/HTTP exporter `0.218.0`, metrics SDK
`2.7.1`, and metrics OTLP/HTTP exporter `0.209.0`.

No live model, provider credential, collector, customer data, or production
deployment was used. Provider/model values were the in-process
`anthropic`/`test-model` fixtures; therefore this refresh proves the repository
contracts, runtime integration, and real local OTLP/HTTP wire boundary, not a
production Anthropic or collector result.

| UTC interval                                      | Exit | Exact command and result                                                                                                                                                                                           |
| ------------------------------------------------- | ---: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 2026-07-29T11:35:57.973Z–2026-07-29T11:35:59.073Z |    0 | `pnpm exec vitest run --config libs/contracts/vitest.config.ts libs/contracts/src/run-protocol/genai-trace.spec.ts --maxWorkers=1` — canonical trace/release contracts and diagnostics passed 4/4                  |
| 2026-07-29T11:36:14.261Z–2026-07-29T11:36:16.112Z |    0 | `(cd libs/shared/tracing && pnpm exec vitest run --maxWorkers=1)` — shared tracing, propagation, privacy rejection, and local OTLP trace/metric wire tests passed 73/73                                            |
| 2026-07-29T11:36:22.603Z–2026-07-29T11:36:24.373Z |    0 | `pnpm exec vitest run --config libs/iris/agents/core/vite.config.ts libs/iris/agents/core/src/agentic/cognition-gateway.test.ts --maxWorkers=1` — governed V6 runtime/provider/tool trace integration passed 11/11 |
| 2026-07-29T11:36:30.077Z–2026-07-29T11:36:31.936Z |    0 | `pnpm exec vitest run --config libs/v6/lilith-agent-welfare/vitest.config.ts --maxWorkers=1` — V6 welfare policy, sanitized telemetry, and local OTLP metric wire tests passed 20/20                               |
| 2026-07-29T11:36:41.921Z–2026-07-29T11:36:56.644Z |    0 | `pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit`                                                                                                                                                       |
| 2026-07-29T11:36:56.646Z–2026-07-29T11:36:59.805Z |    0 | `pnpm exec tsc -p libs/shared/tracing/tsconfig.lib.json --noEmit`                                                                                                                                                  |
| 2026-07-29T11:36:59.807Z–2026-07-29T11:37:03.035Z |    0 | `pnpm exec tsc -p libs/shared/tracing/tsconfig.spec.json --noEmit`                                                                                                                                                 |
| 2026-07-29T11:37:03.037Z–2026-07-29T11:37:12.564Z |    0 | `pnpm exec tsc -p libs/iris/agents/core/tsconfig.lib.json --noEmit`                                                                                                                                                |
| 2026-07-29T11:37:12.565Z–2026-07-29T11:37:14.977Z |    0 | `pnpm exec tsc -p libs/v6/lilith-agent-welfare/tsconfig.lib.json --noEmit`                                                                                                                                         |
| 2026-07-29T11:37:14.979Z–2026-07-29T11:37:17.566Z |    0 | `pnpm exec tsc -p libs/v6/lilith-agent-welfare/tsconfig.spec.json --noEmit`                                                                                                                                        |
| 2026-07-29T11:38:12.367Z–2026-07-29T11:38:29.717Z |    0 | Ten serial executions of the exact cognition command above with `--reporter=dot` — all ten files and 110/110 tests passed, with no source change between attempts                                                  |

The admitted current value suites therefore passed **108/108** once, plus
**110/110** cognition repeatability assertions, and all six applicable
typechecks. Unit, contract, integration, adversarial, and local HTTP-wire
coverage is present:

- the canonical suite rejects content-bearing spans and malformed,
  cross-version, incomplete, duplicate-effect, orphan-artifact, and
  untraceable-release graphs while accepting the exact joined release graph;
- shared tracing tests exercise parent propagation, concurrency, production
  fail-closed configuration, content-bearing metric rejection, and actual local
  OTLP/HTTP trace and metric documents;
- the cognition suite joins live product-runtime, provider, and tool-engine code
  on one common trace and proves untrusted prompt/retrieval/memory fields cannot
  widen authority or leak into spans; and
- the welfare suite exercises both sanitized telemetry generation and a real
  local OTLP/HTTP metric export, including a raw-consent-text non-disclosure
  assertion.

A deployed E2E, production performance/load check, screenshot, media probe,
accessibility run, and UI/mobile/Unreal automation are inapplicable to this
repository-only metadata/runtime boundary. No rendered surface, media artifact,
mobile flow, Unreal content, live provider, deployed collector, or production
traffic changed. Production collector/alert validation remains an external
operations seam and is not replaced by the local HTTP-wire integrations.

### Historical implementation verification

At source publication, the owning packages passed **4,403/4,403** non-duplicated
tests:

| Surface                              |      Result |
| ------------------------------------ | ----------: |
| Canonical contracts full package     | 4,103/4,103 |
| Shared tracing full package          |       72/72 |
| Iris agents-core full package        |     208/208 |
| V6 Lilith agent welfare full package |       20/20 |

The final post-rebase exact-source suites passed **41/41** tests: 4 canonical
trace graph tests, 6 shared tracing/OTLP tests, 11 V6 cognition gateway tests,
and 20 V6 welfare tests. Six applicable typechecks passed (four library and two
spec configs), all four emitted TypeScript builds passed, explicit ESLint over
the 20 changed implementation/test/script files passed with zero warnings,
tracked declarations passed a standalone strict TypeScript check, Prettier
passed, and `git diff --check` passed. The V6 operational verifier passed.

Representative commands:

```text
pnpm exec vitest run --config libs/contracts/vitest.config.ts --maxWorkers=1
pnpm exec vitest run --maxWorkers=1                                  # libs/shared/tracing
pnpm exec vitest run --config libs/iris/agents/core/vite.config.ts --maxWorkers=1
pnpm exec vitest run --config libs/v6/lilith-agent-welfare/vitest.config.ts --maxWorkers=1
pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/shared/tracing/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/shared/tracing/tsconfig.spec.json --noEmit
pnpm exec tsc -p libs/iris/agents/core/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/v6/lilith-agent-welfare/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/v6/lilith-agent-welfare/tsconfig.spec.json --noEmit
pnpm exec tsc -p <each of the four library tsconfigs>                # emitted builds
pnpm exec eslint <20 changed source/test/script files> --no-ignore --max-warnings=0
pnpm exec tsc --ignoreConfig --noEmit --strict --skipLibCheck false <tracked tracing declarations>
pnpm run verify:v6 behavior-welfare-telemetry
pnpm run export:v6:behavior-welfare-telemetry                       # expected fail-closed without endpoint
git diff --check
```

## Bounded verification notes

- During the 2026-07-29 refresh, an initial chained run reported the cognition
  ceiling case as `ok: false` once. A direct runtime reproduction returned the
  expected `max_iterations` result, the exact file immediately passed 11/11, and
  ten further clean-process executions passed 110/110 without a source change.
  No failing condition could be reproduced, so no speculative source or fixture
  edit was made.
- An initial shared-tracing invocation from the repository root exited 1 because
  that package's relative `src/**/*.spec.ts` include was evaluated from the
  wrong root and found no files. The admitted command runs from
  `libs/shared/tracing` and passed 73/73. This was a command-location error, not
  a product/test failure.
- Before the current focused verification, the host had 15 GiB RAM, 31 GiB swap,
  11 GiB available memory, and 16 GiB free disk. An unrelated checkout's
  long-running Vitest worker and this checkout's Nx daemon were active, so no
  broad workspace matrix was started; all OBS-owned checks ran serially.
- The host had 15 GiB RAM and 31 GiB swap. Broad checks ran serially and both
  full contracts runs were polled at intervals below 30 seconds. Available
  memory remained above the 2 GiB stop threshold.
- Full contracts verification emitted the existing missing
  `libs/contracts/src/index.js.map` warning and passed: 181 files and 4,103
  tests.
- Contracts spec-wide TypeScript remains blocked by numerous pre-existing
  unrelated fixture errors (readonly-array incompatibilities, stale object
  shapes, and unused imports). Agents-core spec-wide TypeScript likewise has
  pre-existing unrelated generic-handler/fixture errors. Neither diagnostic set
  names the new OBS-001 source or tests. Both libraries typecheck and build;
  shared tracing and V6 welfare spec typechecks pass.
- The first normal commit attempt ran lint-staged successfully, then an
  unbounded hook unexpectedly launched typechecking for 284 projects at
  `--parallel=4`. It was stopped immediately under the repository's small-host
  safety rule. The source was committed with hooks disabled only after all
  relevant targeted/full tests, typechecks, builds, lint, formatting, and diff
  checks had passed.
- `origin/main` moved through three unrelated V10 commits. The only conflict was
  `pnpm-lock.yaml`; the resolution retained the remote lock exactly plus the
  four OBS-001 importer dependency additions. All 41 focused tests passed again
  after rebase before publication.
- The pre-existing worktree `pnpm-lock.yaml` edit was preserved and restored
  byte-for-byte as Git blob `a37431842e330b3a06b030947c25bebdc72bc2b1`.
  Generated Cargo locks and `libs/urania/` were not staged.
- No customer UI, PWA, admin, mobile, Unreal, or visual behavior changed.
  Frontend skill, Playwright, mobile automation, screenshots, and Unreal
  verification were inapplicable.

## Artifact identity

| Artifact                                       | SHA-256                                                            |
| ---------------------------------------------- | ------------------------------------------------------------------ |
| Canonical cross-version trace contract/auditor | `7ecd2ff521dd66f828bdab7f820e21072bc145a4f25ce99f657ac23245a3a8aa` |
| Shared GenAI span helper                       | `ef1d6b4b023436db3a8060c5df7543d1b00a4c7bef568d9396efa5c1ccb8f42c` |
| Shared tracer implementation                   | `8673bf7dc5322f4d523a9a39e9a0a30b5ef588289439301c35fce44137216579` |
| Shared OTLP metric exporter                    | `eb33bb78b77467b12c277c7531f9f6727f6952550eb5d1de99efd91afd48b3fb` |
| Governed V6 run manager                        | `1c8d89645d9681a1eca5198b9702e6f55589a61df4332654e8e94cc0e7eae2ed` |
| Provider/tool agentic loop                     | `bb0adaa301fd3a7909051e003477d17455814da28eb9ad7396c10a8f88f2b1d2` |
| V6 welfare export surface                      | `8d28ed406750e54d2b9492754e15e8f242649a0c02ac242d108ec2d562aa270c` |
| V6 operational exporter command                | `26e62314cc3338d6835fdd9c7187ac8a078c8637c6795bb0a8487444c8932d47` |

These hashes identify current source at the admitted refresh commit, including
later governance and durability changes that invalidated four original
implementation hashes. The governing contract versions are GenAI trace schema 1,
OpenTelemetry GenAI semantic-convention baseline 1.41.0, and RUN-001 schema and
identity version 1. OBS-001 changes no canon, release policy, quality rubric,
source corpus, model, or media version.

## Release, human-authority, and generated-status disposition

- The repository release verdict is `repository-complete / closed` for OBS-001.
  Runtime placement evidence is the invoked V6 cognition integration, local
  OTLP/HTTP collector tests, and explicit V6 welfare exporter command. There is
  no deployed collector, production trace sample, production alert result, or
  production placement claim.
- No OBS-001 repository contract or test policy requires a signed human release
  decision. No reviewer identity or signature is fabricated. A production
  telemetry deployment remains subject to the operator, privacy, security, and
  release authorities of that environment.
- No OBS-001 external implementation blocker remains. Collector credentials,
  retention/storage, dashboards, alerts, deployment resource attributes, and
  live alert outcomes remain explicitly external operational work.
- No generated capability/status artifact advertises a deployed observability
  result. The completed-work-package admission verifier derives only the
  existence and task identity of this evidence pack from the checked ledger; it
  does not synthesize a production-green claim. No generated status view was
  hand-edited for this refresh.

## Scope and remaining seams

- Production must provide a reachable collector, service identity, credentials,
  deployment-specific resource attributes, storage, retention, dashboards, and
  alerts. Their absence is not represented as deployed evidence.
- The completeness evaluator consumes normalized exported span/artifact/release
  facts. Collector-side normalization and scheduled alert execution belong to
  the deployment/operations plane; malformed or incomplete facts fail this
  repository evaluator rather than being counted as healthy.
- This task defaults the new GenAI surface to metadata-only fields. Portfolio
  collection purpose, retention, deletion, legal hold, tenant isolation,
  break-glass access, and incident policy remain the explicit scope of
  `PRIV-001`.

## Publication identity

- **Source commit:** `f2a69ad905f3704ccc83e80b460dd8ee4ee30dad`
- **Source publication tip:** `f2a69ad905f3704ccc83e80b460dd8ee4ee30dad`
- **Closure evidence commit:** `986b2e263538a3b1eb7ec37f1a32641bec4d1a76`
- **Current refresh source commit:** `b45f4cb6ae5439091f8da2a44980210422b9319c`
- **Current refresh evidence commit:** this evidence-only publication commit;
  the immutable identity is the Git commit containing this file.
