# Telemetry Data Pipeline

Phase 72.5.1.5 adds `@v2/telemetry-data-pipeline`, the server-side path from
`v2-telemetry-firehose` to `warehouse.v2_telemetry.events`.

## Stream Source

The pipeline consumes `TelemetryStreamEnvelope` messages from Kafka or Kinesis.
Each envelope carries the stream kind, `v2-telemetry-firehose` stream name,
partition key, offset, enqueue timestamp, and validated `TelemetryEventBatch`
payload from `@v2/telemetry-ingestion`.

`MemoryTelemetryPipelineSource` is available for tests and local development.
Production deployments provide Kafka or Kinesis adapters that implement
`TelemetryPipelineSource`.

## Transform

`runTelemetryDataPipelineOnce` pulls messages, calls
`validateTelemetryEventBatch`, transforms events with
`transformTelemetryBatchToWarehouseRows`, and acknowledges the source only after
warehouse writes succeed. Invalid batches are sent to the source dead-letter
path before any warehouse write.

Rows use transform schema version `1` and include event IDs, batch IDs,
canonical topics, schema versions, session IDs, source offsets, ingest/transform
timestamps, stable JSON payloads, `payloadSha256`, and `accountIdHash` when
present. `stripForbiddenPayloadFields` removes raw account IDs, email, IP
address, platform user IDs, and raw log lines from transformed payloads.

## Warehouse

The governed destination is `warehouse.v2_telemetry.events`. The package models
ClickHouse, BigQuery, and Redshift targets through `TelemetryWarehouseSink`.
`MemoryTelemetryWarehouseSink` is for local tests; production deployments bind
the same row contract to the selected warehouse target.

Targeted verification:

```bash
pnpm --filter @v2/telemetry-data-pipeline run typecheck
pnpm --filter @v2/telemetry-data-pipeline run test
python V2/ue/Tools/check-v2-telemetry-data-pipeline.py
```