Phase 72.5.1.5 adds @v2/telemetry-data-pipeline, the server-side path from
v2-telemetry-firehose to warehouse.v2_telemetry.events.
Stream Source#
The pipeline consumes TelemetryStreamEnvelope messages from Kafka or Kinesis.
Each envelope carries the stream kind, v2-telemetry-firehose stream name,
partition key, offset, enqueue timestamp, and validated TelemetryEventBatch
payload from @v2/telemetry-ingestion.
MemoryTelemetryPipelineSource is available for tests and local development.
Production deployments provide Kafka or Kinesis adapters that implement
TelemetryPipelineSource.
Transform#
runTelemetryDataPipelineOnce pulls messages, calls
validateTelemetryEventBatch, transforms events with
transformTelemetryBatchToWarehouseRows, and acknowledges the source only after
warehouse writes succeed. Invalid batches are sent to the source dead-letter
path before any warehouse write.
Rows use transform schema version 1 and include event IDs, batch IDs,
canonical topics, schema versions, session IDs, source offsets, ingest/transform
timestamps, stable JSON payloads, payloadSha256, and accountIdHash when
present. stripForbiddenPayloadFields removes raw account IDs, email, IP
address, platform user IDs, and raw log lines from transformed payloads.
Warehouse#
The governed destination is warehouse.v2_telemetry.events. The package models
ClickHouse, BigQuery, and Redshift targets through TelemetryWarehouseSink.
MemoryTelemetryWarehouseSink is for local tests; production deployments bind
the same row contract to the selected warehouse target.
Targeted verification:
pnpm --filter @v2/telemetry-data-pipeline run typecheck
pnpm --filter @v2/telemetry-data-pipeline run test
python V2/ue/Tools/check-v2-telemetry-data-pipeline.py