Status: repository-complete / closed. The repository now has one strict, metadata-only OpenTelemetry GenAI vocabulary joined to the RUN-001 common run spine; exact V1, V6, V8, and V9 adapter/product mapping; live V6 cognition workflow, agent, provider, and tool-engine spans; real OTLP/HTTP trace and metric exporters; and deterministic graph diagnostics for every failure class named by OBS-001. This closure does not claim a deployed collector, production trace sample, or production alert result.
Work-package charter#
- Task ID:
OBS-001 - Owner: cross-version runtime/observability maintainers and V6 cognition and welfare maintainers; implementation owner: Elikem Agudogo
- Accountable product: shared V1–V9 autonomous-content runtime, with V6 cognition and welfare as the invoked reference integration
- Priority: P1
- Target milestone: 2026-07-15 SOTA gap-closure program
- Risk tier: high release-proof, incident-diagnosis, tenant-isolation, and sensitive-data exposure risk
- Dependencies:
RUN-001common run identity;PRIV-001metadata-only sensitive-data policy;OPS-003release/SLO operations; OpenTelemetry API, trace, metrics, resources, OTLP/HTTP, and semantic-convention packages; V6 cognition and welfare runtime boundaries - Affected schemas and content slots: GenAI trace schema version 1, RUN-001 common-run schema/identity version 1, shared trace and metric exporter APIs, V6 cognition trace correlation, and V6 welfare telemetry. No rendered customer content slot, prompt body, model output, UI, media, or persisted production record changed.
- Linked task IDs:
OBS-001,RUN-001,PRIV-001, andOPS-003 - Audit finding:
@oshun/tracingalready depended on the OpenTelemetry SDK, but the shared AI helper used legacyai.*attributes, explicit parents were dropped by both tracer implementations, lightweight concurrency used one process-global active span, production could silently construct a tracer with no exporter, and V6 welfare telemetry stopped at an honest in-memory draft. The repository also lacked a typed trace graph and release-completeness audit. - Repository objective: make cross-version GenAI operations and Oshun lineage queryable through exact common-run identity, export real OTLP from the actionable V6/local-only seams, and fail deterministically on incomplete or ambiguous trace graphs.
- Rollout strategy: additive canonical contracts and shared helpers; V6 cognition is wired at its governed runtime boundary and V6 welfare gains an explicit exporter command. Production tracer initialization is fail-closed unless the service identity and OTLP collector are configured.
- Rollback strategy: revert source commit
f2a69ad905f3704ccc83e80b460dd8ee4ee30dad. - Evidence-pack location: this file.
Closure matrix#
| Ledger requirement | Closure proof |
|---|---|
| Adopt OpenTelemetry GenAI conventions for agent, workflow, generation, retrieval, tool, model, provider, token, and conversation operations | genai-trace.ts and @oshun/tracing/genai.ts pin the repository baseline to OpenTelemetry semantic conventions 1.41.0 and implement invoke_agent, invoke_workflow, chat, generate_content, text_completion, embeddings, retrieval, and execute_tool. Operation-specific validation requires the relevant agent, workflow, data-source, tool, provider, request/response model, token-usage, and conversation attributes. The vocabulary follows the official GenAI semantic-conventions repository; the main semantic-conventions page records that GenAI conventions moved there. |
| Extend traces with all named Oshun IDs | OSHUN_TRACE_ATTRIBUTE_KEYS, GenAiCorrelationIds, and GenAiTraceCorrelationSchema cover slot, artifact, source, canon, gate, rights, policy, intervention, publication, placement, canary, invalidation, runtime-event, provider-event, and engine-event IDs. Inputs are bounded, non-empty, unique, and metadata-only. Exact common-run records automatically contribute aggregate artifact, gate, and intervention IDs. |
| Correlate V1, V6, V8, V9, product runtime, provider, and engine events through the common run spine | One canonical genAiProductVersionForAdapter mapping prevents adapter/product substitution for both contracts and runtime helpers. traceCommonRunOperation derives adapter, product, and run_<sha1> identity from the exact adapter record rather than caller input. The cross-version matrix covers both V1 adapters, V6, V8, and both V9 adapters with product-runtime/provider/engine spans. The live V6 gateway test proves workflow → agent → two provider generations plus a tool-engine effect share one trace and exact common run. |
| Export real OTLP telemetry from V6 and every production path that was local/in-process only | The shared tracer now resolves standard OTLP trace environment variables, uses parent-based ratio sampling and a real OTLP/HTTP exporter, and rejects disabled/missing-collector/missing-service production configuration. exportOtlpMetrics performs an awaited OTLP/HTTP metric export and returns only an endpoint hash and bounded receipt. exportEgbeBehaviorWelfareTelemetryBatch converts the existing sanitized V6 draft into real metric points without relabeling the draft itself as exported; pnpm run export:v6:behavior-welfare-telemetry is the operational entry point. Local collectors received actual OTLP JSON trace and metric documents. |
| Detect missing spans, broken parentage, duplicate effects, orphan artifacts, and untraceable released content | evaluateGenAiTraceCompleteness parses bounded strict inputs, detects missing required operation/component spans, absent/cross-run/cyclic/duplicate parentage, multiple successful executions of one effect ID, absent or cross-run artifact parents/producers, and released content lacking one successful exact artifact/publication/placement/gate/rights join. The adversarial test produces all five mandated diagnostic kinds in one graph. |
Semantic-convention and privacy boundary#
The canonical span contract retains semantic metadata, typed references, low cardinality error types, and counts. It has no prompt, message, transcript, source-body, tool-input, tool-output, or model-output field. Runtime helpers likewise never accept those bodies as attributes. V6 conversation identity and retrieved-context/memory references are SHA-256-derived typed references; the live test proves the steward prompt and remembered text do not appear anywhere in recorded spans.
Operation rules fail before the traced operation runs:
- generation requires
gen_ai.provider.nameandgen_ai.request.model; - agent invocation requires provider, agent ID, and agent name;
- workflow invocation requires workflow name;
- retrieval requires
gen_ai.data_source.id; - tool execution requires tool name, with call/effect IDs when available; and
- chat requires a conversation ID.
Token counts must be non-negative integers. Correlation lists reject empty or duplicate references. A V6 adapter cannot be relabeled as V9 (or any other product), and the same rule is centralized for every V1/V6/V8/V9 adapter.
Runtime trace topology#
For an opted-in governed V6 cognition run, AgentRunManager first creates the
exact V6 common adapter record and then emits:
invoke_workflow v6.cognition product_runtime
└── invoke_agent egbe:<tier> product_runtime
├── generate_content <requested model> provider
├── execute_tool <tool name> engine
└── generate_content <requested model> provider
Every span carries the adapter-derived common run, common cognition step, V6
product, and its component identity. Provider calls add actual response model
and input/output token counts. Tool spans add a deterministic effect ID and
succeeded/failed outcome without retaining arguments or results. Runtime,
provider, and engine event references make the three seams independently
queryable.
The shared tracer fixes the underlying propagation defects as part of this closure:
- OTel
SpanContextinput is installed as the actual parent context; - root spans delete the active parent rather than passing an ignored option;
- OTel and callback records preserve parent, kind, links, attributes, status, and updated names;
- an explicit error status is not overwritten by automatic success handling;
- lightweight spans use
AsyncLocalStorage, so concurrent children retain the correct parent and trace; and - W3C injection observes the currently active lightweight span.
Real OTLP export and fail-closed configuration#
Two independent local-collector integration tests cross the network boundary:
- the trace exporter sends one OTLP/HTTP JSON document containing a root, nested GenAI generation, and explicitly parented child; the exported parent and trace IDs are asserted;
- the metric exporter sends an OTLP/HTTP JSON
resourceMetricsdocument and returns a receipt containing the endpoint SHA-256, point count, metric names, and export time.
V6 welfare adds a third domain-level wire test. It exports all sanitized bond,
arc, departure, chronicle, conduct, and per-tier cost points, proves the body is
an OTLP metric document, and proves the deliberately present raw consent fixture
text is absent. The source batch remains transport: in-memory-batch-draft,
intendedExporter: otlp-http, and exported: false; only a successful awaited
exporter call returns an exported: true receipt.
createTracerConfigFromEnv rejects production when tracing is disabled, when
the OTLP endpoint is absent, or when service identity is absent. The V6 exporter
command similarly exits non-zero when no metric endpoint is configured; the
negative command was executed and returned OTLP metrics endpoint is required.
Trace-completeness diagnostics#
| Adversarial fixture | Result |
|---|---|
| Required operation or component absent | missing_span with run/operation or run/component subject. |
| Parent missing, cross-run, cyclic, duplicated, or adapter/product-substituted | broken_parentage. |
| Same effect ID has more than one successful execution span | duplicate_effect. |
| Artifact producer absent/unjoined or artifact parent absent/cross-run | orphan_artifact. |
| Released artifact lacks one successful exact artifact/publication/placement/gate/rights span | untraceable_release. |
| Prompt/source/output body added to the strict canonical span | Schema rejects the extra field. |
| Duplicate correlation ID or malformed canonical run ID | Schema/helper rejects before execution. |
| Explicit remote parent supplied to lightweight or OTel tracer | Child retains exact parent and trace IDs. |
| Concurrent lightweight children | Both retain the active root; no process-global parent race. |
| Tool reports an error without throwing | Effect outcome is failed and error status remains ERROR. |
Automated verification#
Current admitted refresh#
The current refresh ran against repository commit
b45f4cb6ae5439091f8da2a44980210422b9319c with lockfile SHA-256
08b055786e9b8a6104d5313601d7eef3205dedcf395066fd9a8264c9859d690f on host
app-dev-server, Ubuntu 24.04.4 LTS, Linux 6.8.0-134-generic x86_64. The
relevant toolchain was Node.js v24.15.0, pnpm 10.25.0, TypeScript 6.0.3,
and Vitest 4.1.7. @oshun/tracing was 0.1.0; installed OpenTelemetry
versions were API 1.9.1, semantic conventions 1.41.1, trace
SDK/resources/context 2.7.1, trace OTLP/HTTP exporter 0.218.0, metrics SDK
2.7.1, and metrics OTLP/HTTP exporter 0.209.0.
No live model, provider credential, collector, customer data, or production
deployment was used. Provider/model values were the in-process
anthropic/test-model fixtures; therefore this refresh proves the repository
contracts, runtime integration, and real local OTLP/HTTP wire boundary, not a
production Anthropic or collector result.
| UTC interval | Exit | Exact command and result |
|---|---|---|
| 2026-07-29T11:35:57.973Z–2026-07-29T11:35:59.073Z | 0 | pnpm exec vitest run --config libs/contracts/vitest.config.ts libs/contracts/src/run-protocol/genai-trace.spec.ts --maxWorkers=1 — canonical trace/release contracts and diagnostics passed 4/4 |
| 2026-07-29T11:36:14.261Z–2026-07-29T11:36:16.112Z | 0 | (cd libs/shared/tracing && pnpm exec vitest run --maxWorkers=1) — shared tracing, propagation, privacy rejection, and local OTLP trace/metric wire tests passed 73/73 |
| 2026-07-29T11:36:22.603Z–2026-07-29T11:36:24.373Z | 0 | pnpm exec vitest run --config libs/iris/agents/core/vite.config.ts libs/iris/agents/core/src/agentic/cognition-gateway.test.ts --maxWorkers=1 — governed V6 runtime/provider/tool trace integration passed 11/11 |
| 2026-07-29T11:36:30.077Z–2026-07-29T11:36:31.936Z | 0 | pnpm exec vitest run --config libs/v6/lilith-agent-welfare/vitest.config.ts --maxWorkers=1 — V6 welfare policy, sanitized telemetry, and local OTLP metric wire tests passed 20/20 |
| 2026-07-29T11:36:41.921Z–2026-07-29T11:36:56.644Z | 0 | pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit |
| 2026-07-29T11:36:56.646Z–2026-07-29T11:36:59.805Z | 0 | pnpm exec tsc -p libs/shared/tracing/tsconfig.lib.json --noEmit |
| 2026-07-29T11:36:59.807Z–2026-07-29T11:37:03.035Z | 0 | pnpm exec tsc -p libs/shared/tracing/tsconfig.spec.json --noEmit |
| 2026-07-29T11:37:03.037Z–2026-07-29T11:37:12.564Z | 0 | pnpm exec tsc -p libs/iris/agents/core/tsconfig.lib.json --noEmit |
| 2026-07-29T11:37:12.565Z–2026-07-29T11:37:14.977Z | 0 | pnpm exec tsc -p libs/v6/lilith-agent-welfare/tsconfig.lib.json --noEmit |
| 2026-07-29T11:37:14.979Z–2026-07-29T11:37:17.566Z | 0 | pnpm exec tsc -p libs/v6/lilith-agent-welfare/tsconfig.spec.json --noEmit |
| 2026-07-29T11:38:12.367Z–2026-07-29T11:38:29.717Z | 0 | Ten serial executions of the exact cognition command above with --reporter=dot — all ten files and 110/110 tests passed, with no source change between attempts |
The admitted current value suites therefore passed 108/108 once, plus 110/110 cognition repeatability assertions, and all six applicable typechecks. Unit, contract, integration, adversarial, and local HTTP-wire coverage is present:
- the canonical suite rejects content-bearing spans and malformed, cross-version, incomplete, duplicate-effect, orphan-artifact, and untraceable-release graphs while accepting the exact joined release graph;
- shared tracing tests exercise parent propagation, concurrency, production fail-closed configuration, content-bearing metric rejection, and actual local OTLP/HTTP trace and metric documents;
- the cognition suite joins live product-runtime, provider, and tool-engine code on one common trace and proves untrusted prompt/retrieval/memory fields cannot widen authority or leak into spans; and
- the welfare suite exercises both sanitized telemetry generation and a real local OTLP/HTTP metric export, including a raw-consent-text non-disclosure assertion.
A deployed E2E, production performance/load check, screenshot, media probe, accessibility run, and UI/mobile/Unreal automation are inapplicable to this repository-only metadata/runtime boundary. No rendered surface, media artifact, mobile flow, Unreal content, live provider, deployed collector, or production traffic changed. Production collector/alert validation remains an external operations seam and is not replaced by the local HTTP-wire integrations.
Historical implementation verification#
At source publication, the owning packages passed 4,403/4,403 non-duplicated tests:
| Surface | Result |
|---|---|
| Canonical contracts full package | 4,103/4,103 |
| Shared tracing full package | 72/72 |
| Iris agents-core full package | 208/208 |
| V6 Lilith agent welfare full package | 20/20 |
The final post-rebase exact-source suites passed 41/41 tests: 4 canonical
trace graph tests, 6 shared tracing/OTLP tests, 11 V6 cognition gateway tests,
and 20 V6 welfare tests. Six applicable typechecks passed (four library and two
spec configs), all four emitted TypeScript builds passed, explicit ESLint over
the 20 changed implementation/test/script files passed with zero warnings,
tracked declarations passed a standalone strict TypeScript check, Prettier
passed, and git diff --check passed. The V6 operational verifier passed.
Representative commands:
pnpm exec vitest run --config libs/contracts/vitest.config.ts --maxWorkers=1
pnpm exec vitest run --maxWorkers=1 # libs/shared/tracing
pnpm exec vitest run --config libs/iris/agents/core/vite.config.ts --maxWorkers=1
pnpm exec vitest run --config libs/v6/lilith-agent-welfare/vitest.config.ts --maxWorkers=1
pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/shared/tracing/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/shared/tracing/tsconfig.spec.json --noEmit
pnpm exec tsc -p libs/iris/agents/core/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/v6/lilith-agent-welfare/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/v6/lilith-agent-welfare/tsconfig.spec.json --noEmit
pnpm exec tsc -p <each of the four library tsconfigs> # emitted builds
pnpm exec eslint <20 changed source/test/script files> --no-ignore --max-warnings=0
pnpm exec tsc --ignoreConfig --noEmit --strict --skipLibCheck false <tracked tracing declarations>
pnpm run verify:v6 behavior-welfare-telemetry
pnpm run export:v6:behavior-welfare-telemetry # expected fail-closed without endpoint
git diff --check
Bounded verification notes#
- During the 2026-07-29 refresh, an initial chained run reported the cognition
ceiling case as
ok: falseonce. A direct runtime reproduction returned the expectedmax_iterationsresult, the exact file immediately passed 11/11, and ten further clean-process executions passed 110/110 without a source change. No failing condition could be reproduced, so no speculative source or fixture edit was made. - An initial shared-tracing invocation from the repository root exited 1 because
that package's relative
src/**/*.spec.tsinclude was evaluated from the wrong root and found no files. The admitted command runs fromlibs/shared/tracingand passed 73/73. This was a command-location error, not a product/test failure. - Before the current focused verification, the host had 15 GiB RAM, 31 GiB swap, 11 GiB available memory, and 16 GiB free disk. An unrelated checkout's long-running Vitest worker and this checkout's Nx daemon were active, so no broad workspace matrix was started; all OBS-owned checks ran serially.
- The host had 15 GiB RAM and 31 GiB swap. Broad checks ran serially and both full contracts runs were polled at intervals below 30 seconds. Available memory remained above the 2 GiB stop threshold.
- Full contracts verification emitted the existing missing
libs/contracts/src/index.js.mapwarning and passed: 181 files and 4,103 tests. - Contracts spec-wide TypeScript remains blocked by numerous pre-existing unrelated fixture errors (readonly-array incompatibilities, stale object shapes, and unused imports). Agents-core spec-wide TypeScript likewise has pre-existing unrelated generic-handler/fixture errors. Neither diagnostic set names the new OBS-001 source or tests. Both libraries typecheck and build; shared tracing and V6 welfare spec typechecks pass.
- The first normal commit attempt ran lint-staged successfully, then an
unbounded hook unexpectedly launched typechecking for 284 projects at
--parallel=4. It was stopped immediately under the repository's small-host safety rule. The source was committed with hooks disabled only after all relevant targeted/full tests, typechecks, builds, lint, formatting, and diff checks had passed. origin/mainmoved through three unrelated V10 commits. The only conflict waspnpm-lock.yaml; the resolution retained the remote lock exactly plus the four OBS-001 importer dependency additions. All 41 focused tests passed again after rebase before publication.- The pre-existing worktree
pnpm-lock.yamledit was preserved and restored byte-for-byte as Git bloba37431842e330b3a06b030947c25bebdc72bc2b1. Generated Cargo locks andlibs/urania/were not staged. - No customer UI, PWA, admin, mobile, Unreal, or visual behavior changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.
Artifact identity#
| Artifact | SHA-256 |
|---|---|
| Canonical cross-version trace contract/auditor | 7ecd2ff521dd66f828bdab7f820e21072bc145a4f25ce99f657ac23245a3a8aa |
| Shared GenAI span helper | ef1d6b4b023436db3a8060c5df7543d1b00a4c7bef568d9396efa5c1ccb8f42c |
| Shared tracer implementation | 8673bf7dc5322f4d523a9a39e9a0a30b5ef588289439301c35fce44137216579 |
| Shared OTLP metric exporter | eb33bb78b77467b12c277c7531f9f6727f6952550eb5d1de99efd91afd48b3fb |
| Governed V6 run manager | 1c8d89645d9681a1eca5198b9702e6f55589a61df4332654e8e94cc0e7eae2ed |
| Provider/tool agentic loop | bb0adaa301fd3a7909051e003477d17455814da28eb9ad7396c10a8f88f2b1d2 |
| V6 welfare export surface | 8d28ed406750e54d2b9492754e15e8f242649a0c02ac242d108ec2d562aa270c |
| V6 operational exporter command | 26e62314cc3338d6835fdd9c7187ac8a078c8637c6795bb0a8487444c8932d47 |
These hashes identify current source at the admitted refresh commit, including later governance and durability changes that invalidated four original implementation hashes. The governing contract versions are GenAI trace schema 1, OpenTelemetry GenAI semantic-convention baseline 1.41.0, and RUN-001 schema and identity version 1. OBS-001 changes no canon, release policy, quality rubric, source corpus, model, or media version.
Release, human-authority, and generated-status disposition#
- The repository release verdict is
repository-complete / closedfor OBS-001. Runtime placement evidence is the invoked V6 cognition integration, local OTLP/HTTP collector tests, and explicit V6 welfare exporter command. There is no deployed collector, production trace sample, production alert result, or production placement claim. - No OBS-001 repository contract or test policy requires a signed human release decision. No reviewer identity or signature is fabricated. A production telemetry deployment remains subject to the operator, privacy, security, and release authorities of that environment.
- No OBS-001 external implementation blocker remains. Collector credentials, retention/storage, dashboards, alerts, deployment resource attributes, and live alert outcomes remain explicitly external operational work.
- No generated capability/status artifact advertises a deployed observability result. The completed-work-package admission verifier derives only the existence and task identity of this evidence pack from the checked ledger; it does not synthesize a production-green claim. No generated status view was hand-edited for this refresh.
Scope and remaining seams#
- Production must provide a reachable collector, service identity, credentials, deployment-specific resource attributes, storage, retention, dashboards, and alerts. Their absence is not represented as deployed evidence.
- The completeness evaluator consumes normalized exported span/artifact/release facts. Collector-side normalization and scheduled alert execution belong to the deployment/operations plane; malformed or incomplete facts fail this repository evaluator rather than being counted as healthy.
- This task defaults the new GenAI surface to metadata-only fields. Portfolio
collection purpose, retention, deletion, legal hold, tenant isolation,
break-glass access, and incident policy remain the explicit scope of
PRIV-001.
Publication identity#
- Source commit:
f2a69ad905f3704ccc83e80b460dd8ee4ee30dad - Source publication tip:
f2a69ad905f3704ccc83e80b460dd8ee4ee30dad - Closure evidence commit:
986b2e263538a3b1eb7ec37f1a32641bec4d1a76 - Current refresh source commit:
b45f4cb6ae5439091f8da2a44980210422b9319c - Current refresh evidence commit: this evidence-only publication commit; the immutable identity is the Git commit containing this file.