# Maya Shader Pipeline

This document is the engineering source of truth for the Maya renderer shader
pipeline in V2. It connects renderer-owned WGSL sources, permutation planning,
cross-compilation, cache generation, diagnostics, complexity analysis, usage
tracking, parallel compilation, and binary size regression gates.

## Source Of Truth

The implementation lives in `libs/maya/engine-core/crates/maya-renderer/src`.
The pipeline uses WGSL-primary shader sources and stable metadata from these
contracts:

- `V2/ue/Build/Shaders/v2-maya-shader-source-management.json`
- `V2/ue/Build/Shaders/v2-maya-shader-permutation-system.json`
- `V2/ue/Build/Shaders/v2-maya-shader-permutation-explosion-management.json`
- `V2/ue/Build/Shaders/v2-maya-shader-cross-compilation-pipeline.json`
- `V2/ue/Build/Shaders/v2-maya-shader-pso-caching.json`
- `V2/ue/Build/Shaders/v2-maya-shader-precompilation-cook.json`
- `V2/ue/Build/Shaders/v2-maya-shader-hot-reload.json`
- `V2/ue/Build/Shaders/v2-maya-shader-include-module-system.json`
- `V2/ue/Build/Shaders/v2-maya-shader-debugging-symbols.json`
- `V2/ue/Build/Shaders/v2-maya-shader-complexity-analysis.json`
- `V2/ue/Build/Shaders/v2-maya-shader-variant-usage-tracking.json`
- `V2/ue/Build/Shaders/v2-maya-shader-compilation-error-reporting.json`
- `V2/ue/Build/Shaders/v2-maya-shader-compilation-parallelization.json`
- `V2/ue/Build/Shaders/v2-maya-shader-binary-size-tracking.json`
- `V2/ue/Build/Shaders/v2-shader-distribution.json`

The required CI validator for this document is:

```bash
python3 V2/ue/Tools/check-v2-maya-shader-pipeline-documentation.py
```

## Architecture

The pipeline has one ordering rule: source identity is decided before
permutation identity, permutation identity is decided before target compilation,
and target compilation is decided before cache, size, and release gates.

| Stage                     | Owner                                                       | Primary artifact                      |
| ------------------------- | ----------------------------------------------------------- | ------------------------------------- |
| Source catalog            | `shader_source_management`                                  | WGSL descriptors under `shaders/maya` |
| Include composition       | `shader_include_module_system`                              | composed WGSL with line spans         |
| Permutations              | `shader_permutation_system`                                 | `ShaderPermutationManifest`           |
| Pruning                   | `shader_permutation_explosion_management`                   | management plan from usage profile    |
| Cross-compile jobs        | `shader_cross_compilation_pipeline`                         | `ShaderCrossCompileJob` rows          |
| Local and farm scheduling | `shader_compilation_parallelization`                        | local workers and Horde shards        |
| Diagnostics               | `shader_compilation_error_reporting`                        | source-mapped reports                 |
| Debug symbols             | `shader_debugging_symbols`                                  | source maps and symbol packages       |
| PSO and shader caches     | `shader_cache`, `pipeline_state_object_cache`               | content-addressed blobs               |
| Cook and distribution     | `shader_precompilation_cook`, shader distribution plan      | DDC fill outputs                      |
| Quality gates             | `shader_complexity_analysis`, `shader_binary_size_tracking` | complexity and size reports           |

The core Maya shader library currently expands to 32 variants and 96 default
cross-compile jobs for Vulkan SPIR-V, DX12 DXIL, and Metal MSL. The release
target shorthand is Vulkan SPIR-V DX12 DXIL Metal MSL.

## Source And Include Authoring

All new renderer-owned shader source starts as WGSL. Add a
`ShaderSourceDescriptor` with a stable `maya.<feature>.<name>` id, a relative
path under `shaders/maya/<feature>/`, a concrete `ShaderStage`, and an entry
point that exists in the WGSL source.

Reusable code belongs in include modules when it is shared across features or
would otherwise duplicate math, binding, or lighting logic. Include modules must
keep source spans intact so include source mapping can let
`ShaderComposedSource` map generated compiler diagnostics back to original
shader files and include modules.

Do not add backend-specific source as the canonical file. HLSL, MSL, DXIL, and
SPIR-V are generated artifacts produced by the cross-compilation pipeline and
validated by the target contracts.

## Permutations And Usage

Permutation dimensions must describe real runtime choices such as skin, damage,
sweat, cloth, hair, lighting, shadow quality, reflection, volumetric, or
post-process features. A new dimension must have:

- a stable toggle name and bit assignment,
- a default state,
- a content or runtime owner,
- a pruning rule in the explosion manager when the dimension can multiply many
  variants,
- usage telemetry through `shader_variant_usage_tracking` when runtime evidence
  can prove which variants are used.

The pruning loop is evidence-driven. Runtime profiles feed
`ShaderPermutationUsageProfile`; the explosion manager keeps used variants,
prunes unused variants, and protects required platform or content variants.

## Compilation And Distribution

`shader_cross_compilation_pipeline` converts each permutation into deterministic
jobs for `ShaderCrossCompileTarget::VulkanSpirV`,
`ShaderCrossCompileTarget::Dx12Dxil`, and `ShaderCrossCompileTarget::MetalMsl`.
Each job carries source metadata, defines, target key, compiler version, stage
list, debug-symbol setting, and line-map preservation setting.

`shader_compilation_parallelization` schedules those jobs across local CPU
workers and emits build-farm shards grouped by target and Horde pool. Local
execution uses scoped worker threads and restores deterministic output ordering.
Farm execution uses the same target/job metadata so local and remote compilers
publish compatible cache artifacts.

The distribution plan in `V2/ue/Build/Shaders/v2-shader-distribution.json` fills
shared DDC through ShaderCompileWorker-backed commandlets, bounded shard sizes,
and platform-specific Horde pools.

## Diagnostics And Debugging

Compiler failures must be routed through `shader_compilation_error_reporting`.
The reporter normalizes backend-specific diagnostics, classifies syntax, type,
symbol, entry point, binding, unsupported-feature, and include-module failures,
and maps generated lines back through include composition spans.

Debug symbol packages must preserve source path, entry point, target, stage, and
line-map metadata. Do not strip line maps from development or CI builds that are
used for shader triage.

When a shader fails CI, the first triage artifact should be the human-readable
error report, followed by the composed source excerpt, debug-symbol package, and
the exact `ShaderCrossCompileJob` metadata.

## Cache Size And Regression Gates

Shader cache quality has two separate gates:

- `shader_complexity_analysis` estimates ALU operations, texture fetches, branch
  counts, and severity for expensive shader variants.
- `shader_binary_size_tracking` records compiled artifact bytes, aggregates
  per-platform shader cache totals, exports baselines, and alerts on total,
  platform, or individual artifact growth.

The default binary-size gates use a 10 percent total cache regression threshold,
a 10 percent per-platform cache regression threshold, a 65536 byte artifact
growth warning threshold, and a 67108864 byte default platform cache budget.

PSO cache keys and shader cache keys are content-addressed. Changing source
code, defines, entry point, backend, or compiler version must produce a new key
and new size telemetry.

## CI And Release Evidence

The V2 CI workflow must validate every shader contract listed in Source Of
Truth. Horde must publish required gates for source management, permutation
generation, permutation pruning, cross-compilation, PSO caching, precompilation,
hot reload, include modules, debugging symbols, complexity, variant usage, error
reporting, parallelization, binary size tracking, and shader distribution.

Release evidence for shader pipeline changes must include:

- the changed contract and checker,
- the relevant Rust test filter when code changes,
- the generated or validated JSON contract,
- CI workflow and Horde gate validation,
- shader distribution evidence when DDC or build-farm behavior changes,
- binary-size baseline comparison when compiled output size can change,
- a TODO evidence entry with the verification commands that passed.

## Verification Commands

Use targeted checks while iterating:

```bash
python3 V2/ue/Tools/check-v2-maya-shader-pipeline-documentation.py
python3 V2/ue/Tools/check-v2-ci-workflow.py
python3 V2/ue/Tools/check-v2-buildgraph.py
python3 V2/tools/validate-v2-docs.py
```

Use Rust filters when implementation modules change:

```bash
CARGO_BUILD_JOBS=1 cargo test --locked --manifest-path libs/maya/engine-core/Cargo.toml -p maya-renderer shader_cross_compilation_pipeline --lib
CARGO_BUILD_JOBS=1 cargo test --locked --manifest-path libs/maya/engine-core/Cargo.toml -p maya-renderer shader_compilation_parallelization --lib
CARGO_BUILD_JOBS=1 cargo test --locked --manifest-path libs/maya/engine-core/Cargo.toml -p maya-renderer shader_binary_size_tracking --lib
```