Eve library

ISIS v2 WS06-010: 3D Pipeline Benchmark Suites

Guide

Revision 1

On this page

Scope#

WS06-010 adds benchmark suites for:

  • Geometry quality validation outcomes.
  • Texture fidelity validation outcomes.
  • Downstream engine ingest success validation outcomes.

Research and Reuse#

Implemented by composing existing validated primitives already in ISIS:

  • Geometry checks: validateMeshQuality (mesh-quality-validator.ts)
  • Texture checks: validatePBRTextureCoverage + validateUVAndTexelDensity
  • Engine ingest checks: createEngineExportBundles + validateEngineExportBundles

This approach avoids duplicate logic and keeps benchmark scoring coupled to production validators.

Deliverables#

  1. Added a new benchmark module:
  • libs/isis/3d-generation/src/processing/three-d-benchmark-suite.ts
  1. Added benchmark coverage tests:
  • libs/isis/3d-generation/src/processing/three-d-benchmark-suite.spec.ts
  1. Exported benchmark APIs through processing/root package exports for reuse:
  • libs/isis/3d-generation/src/processing/index.ts
  • libs/isis/3d-generation/src/index.ts

Benchmark Suite Capabilities#

  • Runs deterministic benchmark datasets across geometry, texture, and engine ingest domains.
  • Tracks per-case:
    • Validation result
    • Expected result
    • Expectation accuracy
    • Error/warning counts
    • Weighted score
  • Produces aggregate report with:
    • Suite-level expectation accuracy
    • Suite-level average score
    • Engine ingest success rate
    • Cross-suite overall score and threshold pass/fail

Verification Commands#

bash
pnpm --filter @isis/3d-generation exec vitest run src/processing/three-d-benchmark-suite.spec.ts src/processing/three-d-provenance.spec.ts src/processing/mesh-quality-validator.spec.ts src/processing/pbr-texture-validator.spec.ts src/processing/uv-texel-quality-validator.spec.ts src/processing/engine-export-bundles.spec.ts src/processing/dcc-export-bundles.spec.ts src/processing/lod-pipeline-validator.spec.ts src/processing/rig-readiness-validator.spec.ts
pnpm --filter @isis/3d-generation exec tsc -p tsconfig.json --noEmit
pnpm --filter @isis/workflows exec vitest run src/workflows.spec.ts
pnpm --filter @isis/workflows exec tsc -p tsconfig.json --noEmit
pnpm --filter @isis/generation-api exec vitest run src/schemas/job.schema.spec.ts src/workflows/executor.spec.ts
pnpm --filter @isis/output-registry exec vitest run src/schemas/output.schema.spec.ts
pnpm --filter @isis/generation-api exec tsc -p tsconfig.json --noEmit
pnpm --filter @isis/output-registry exec tsc -p tsconfig.json --noEmit
node scripts/isis/validate-v2-compliance-gate.mjs