# @isis/3d-browser Browser-first 3D viewing, gaussian splatting, editing, and inference infrastructure for Isis. ## Purpose `@isis/3d-browser` is the Phase 71 browser surface for generated 3D assets. The project scaffold is intentionally split into four future-facing modules so the remaining `71.5.x` tasks can land without reorganizing the library: - `viewer`: mesh viewing, PBR inspection, comparison, and progressive streaming - `gaussian-splatting`: WebGPU 3DGS and 4DGS playback - `editing`: mesh sculpting, painting, UV inspection, history, and export - `inference`: browser-side WASM/WebGPU inference and hybrid offload ## Implemented Viewer Surface `71.5.1.2`, `71.5.1.3`, `71.5.1.4`, `71.5.1.5`, `71.5.1.6`, `71.5.1.7`, `71.5.1.8`, `71.5.1.9`, and `71.5.1.10` are now implemented through the WebGPU-native PBR renderer in `src/viewer/pbr-renderer.ts`, the projected wireframe helper in `src/viewer/wireframe-overlay.ts`, the UV atlas helper in `src/viewer/uv-overlay.ts`, and the turntable interaction surface in `src/viewer/turntable-viewer.ts`, plus the reusable scene analysis helper in `src/viewer/mesh-statistics.ts`, the comparison snapshot helper in `src/viewer/comparison-mode.ts`, the AR Quick Look launch helper in `src/viewer/quick-look.ts`, and the progressive LOD-streaming helper in `src/viewer/progressive-streaming.ts`. Together they provide: - physically based direct lighting with GGX shading - diffuse SH environment lighting and prefiltered reflection cube sampling - directional shadow mapping with PCF filtering and normal bias - typed geometry, material, environment, and camera contracts for generated 3D assets - deterministic CPU-side uniform packing and cache-aware resource preparation - scene-wide PBR inspection modes for lit, albedo, normal, roughness, and metallic review without leaving the browser viewer - reusable scene statistics for polygon, vertex, material, texture, bounds, and approximate resource-weight review - headless comparison contracts for single, side-by-side, and overlay review of two generated variants using shared scene statistics and delta summaries - typed AR Quick Look launch-url generation and browser capability assessment for same-origin USDZ review flows on Safari - typed progressive LOD manifests, screen-coverage selection, and async preview-to-full-quality mesh promotion for browser streaming workflows - orbit, pan, wheel zoom, fit-to-scene framing, and optional auto-turntable camera control - reusable UV atlas extraction with island, triangle, coverage, and bounds metadata for browser-side layout inspection - indexed-triangle wireframe projection for mesh topology inspection in browser - viewer orchestration that can drive renderer-backed frames or DOM-hosted QA harnesses from the same camera model ## Implemented Gaussian Splatting Surface `71.5.2.1`, `71.5.2.2`, `71.5.2.3`, `71.5.2.4`, `71.5.2.5`, and `71.5.2.6` are now implemented through the WebGPU 3D Gaussian Splatting surface in `src/gaussian-splatting/renderer.ts`, the shared scene loader in `src/gaussian-splatting/scene-loader.ts`, the shared LOD helper in `src/gaussian-splatting/lod.ts`, the `.igsplat` transport helper in `src/gaussian-splatting/scene-compression.ts`, the shared mesh extraction helper in `src/gaussian-splatting/mesh-extraction.ts`, the shared animation helper in `src/gaussian-splatting/animation.ts`, and the Oshun studio route host in `apps/oshun/web/src/components/studio/StudioIsisGaussianSplattingWorkspace.tsx`. Together they provide: - typed scene, camera, sort, and frame-profile contracts for browser 3DGS review - typed `.ply` and `.splat` scene loading contracts with format detection and URL-fetch helpers - deterministic scene validation and GPU packing for positions, scales, rotations, colors, and opacities - GraphDeco-style PLY parsing for ASCII and binary little-endian payloads, including SH/DC color decode, opacity sigmoid decode, and retained spherical-harmonics coefficients up to degree 3 - `.splat` parsing for both raw 32-byte standard payloads and the repo worker's `ISIS_SPLAT_V1` headered binary payloads - typed `.igsplat` transport compression and decompression with quantized position, scale, rotation, color, opacity, and SH payload packing plus optional raw, gzip, or brotli transport codecs - checksum-validated round-trip decoding and URL fetch helpers for browser-side network transfer review - deterministic browser-side LOD generation with bounds-preserving anchors, selectable reduction strategies, and hysteresis-backed coverage bands - browser-side 4D gaussian clip validation, interpolation, stable LOD-aware clip subsetting, and reusable playback-state control for animated scene review - browser-side gaussian-to-mesh extraction through density voxelization, tetrahedral isosurface recovery, smoothing passes, and mesh-scene conversion - extracted mesh statistics and wireframe-preview surfaces that can feed browser inspection routes without leaving the gaussian workspace - cull-before-sort visibility rejection using clip-space radius estimates - CPU-native and depth-bucket sort strategies for browser-friendly depth ordering - WebGPU billboard expansion that projects rotated Gaussian axes directly in WGSL - premultiplied-alpha Gaussian falloff composition with reusable scene buffers - stable sort reuse across small camera deltas to avoid redundant upload work - browser-route telemetry for ingest status, parser metadata, active LOD level, 4D playback timing, transfer compression status, coverage, culling, and payload size ## Implemented Editing Surface `71.5.3.1`, `71.5.3.2`, `71.5.3.3`, `71.5.3.4`, `71.5.3.5`, `71.5.3.6`, `71.5.3.7`, and `71.5.3.8` are now implemented through the shared sculpting surface in `src/editing/sculpting.ts`, the shared UV-aware texture painting surface in `src/editing/texture-painting.ts`, the shared vertex-painting surface in `src/editing/vertex-painting.ts`, the shared boolean surface in `src/editing/boolean-operations.ts`, the shared UV layout surface in `src/editing/uv-layout.ts`, the shared bounded history surface in `src/editing/history.ts`, the shared export surface in `src/editing/mesh-export.ts`, and the Oshun route host in `apps/oshun/web/src/components/studio/StudioIsis3DEditingWorkspace.tsx`. Together they provide: - typed browser sculpting contracts for push, pull, smooth, and flatten brushes - typed texture-painting contracts for UV-aware world-space strokes, blend modes, upload planning, and sampled texture inspection - typed vertex-painting contracts for per-vertex material assignment, blend modes, interleaved upload planning, and weighted influence inspection - typed browser boolean contracts for union, subtract, and intersect against sphere, box, and cylinder operands with deterministic relief-volume sampling and preserved mesh topology - typed browser UV layout contracts for atlas inspection, packed unwrap preview, manual island translation, scale, rotation, snapping, and texcoord upload telemetry - typed bounded snapshot history contracts for immutable geometry/texture checkpoints, undo depth, redo depth, and reset checkpoints - reusable geometry validation and deterministic topology adjacency generation - stroke preview sampling with weighted influence metadata and affected-vertex accounting - UV-raster paint preview sampling with affected-texel, coverage, and atlas bounds metadata - boolean preview sampling with per-vertex surface deltas, affected-triangle counts, and estimated volume-change telemetry - CPU-backed deformation application that preserves shared mesh contracts for browser QA and fallback runtimes - CPU-backed texture rasterization that paints directly onto browser-resident base-color maps and publishes WebGPU upload-ready layout telemetry - CPU-backed vertex color painting that updates browser-resident geometry color layers and publishes interleaved vertex-buffer upload telemetry - CPU-backed UV layout inspection that reports island bounds, texel density, overlap, out-of-bounds coverage, and packed atlas utilization from any mesh - CPU-backed UV layout application that commits manual texcoord transforms back into browser-resident geometry and emits WebGPU upload-ready stride and byte telemetry - CPU-backed constructive relief booleans that update browser-resident mesh positions and normals while preserving UVs, colors, and triangle topology for downstream sculpt, paint, and export workflows - immutable geometry and texture snapshot cloning for browser-side edit-history branching, replay, and reset restoration - typed multi-format mesh export for GLB, inline glTF, OBJ bundles, FBX bundles, USDA bundles, and USDZ packages emitted directly from the browser editing state - deterministic PNG texture encoding and stored-zip packaging utilities for portable browser-side export bundles - glTF-native export with embedded PBR texture payloads plus OBJ, FBX, and USD interchange handoff documents that preserve the active mesh, UVs, normals, vertex colors, and base-color surface - per-stroke normal recomputation and bounds/displacement telemetry - WebGPU compute shader source generation and dispatch sizing for brush and normal reconstruction passes - a dedicated studio workspace route with shared sculpt, texture-paint, UV-layout, vertex-paint, boolean, and undo/redo history mode switching, live inspection surfaces, preset-driven QA targets, keyboard history shortcuts, shader or upload telemetry, direct multi-format export downloads, and route discovery ## Implemented Inference Surface `71.5.4.1`, `71.5.4.2`, `71.5.4.3`, `71.5.4.4`, and `71.5.4.5` are now implemented through the shared browser SF3D WASM surface in `src/inference/sf3d-wasm.ts`, the shared browser TripoSR WASM surface in `src/inference/triposr-wasm.ts`, the generic ONNX WebGPU backend in `src/inference/webgpu-backend.ts`, the browser model cache in `src/inference/model-cache.ts`, the hybrid browser or server orchestrator in `src/inference/hybrid-inference.ts`, and the module export surface in `src/inference/index.ts`. Together they provide: - typed browser capability assessment for secure-context, SIMD, threaded WASM, SharedArrayBuffer, and IndexedDB-aware SF3D execution planning - typed bundle manifests for ONNX Runtime Web loaders, WASM binaries, and quantized SF3D ORT or ONNX graph artifacts - deterministic browser compile planning for draft, standard, and high-quality image-to-3D presets with runtime-variant, memory, latency, and artifact selection telemetry - foreground-aware RGBA preprocessing that crops around the alpha silhouette, resizes to the selected browser inference resolution, and emits coverage, luma, and fill-ratio metrics - an embedded browser-loadable WASM feature kernel that runs without server dependencies and exposes a stable depth-byte ABI for mesh synthesis - browser-side single-image relief-mesh synthesis that emits closed triangle geometry, recomputed normals, vertex colors, UVs, and a PBR-ready base-color texture handoff - a lighter TripoSR browser compile profile with smaller target resolutions, faster estimated latency budgets, and explicit vertex-color versus baked- texture output planning - browser-side TripoSR mesh synthesis that defaults to vertex-color output for smaller client payloads while still supporting optional baked-texture handoffs that mirror the upstream model surface - typed ONNX Runtime WebGPU backend contracts for capability assessment, session creation, tensor marshaling, graph-optimization selection, and GPU-buffer output preferences - reusable WebGPU execution-provider planning that can bind quantized ONNX or ORT models through the browser `webgpu` execution provider without forcing a hard dependency on `onnxruntime-web` into the shared package surface - reusable IndexedDB-backed browser model caching with explicit artifact bundle contracts, LRU eviction, per-model inspection, and one-time download reuse across browser sessions - reusable artifact acquisition flows that can prefetch selected SF3D or TripoSR runtime artifacts, resolve browser-facing asset URLs, and materialize cached buffers back into inference sessions without redesigning the inference module surface - reusable hybrid image-to-3D planning that weighs browser latency, device memory, network upload cost, remote queue delay, and requested quality to choose browser-only, server-only, or preview-plus-offload execution - a browser-side client for the repo's `/v1/3d` provider contract, including capability discovery, image-to-3D job submission, polling, cancellation, and normalized remote asset descriptors for final GLB or OBJ handoff - unified hybrid orchestration that can emit a fast local preview, offload the final asset to a remote 3D provider when browser execution is constrained, and publish cross-stage progress updates without changing the surrounding app surface ## Frontend Direction - Visual thesis: a cinematic, studio-grade browser workspace with the canvas as the primary surface and controls treated as supporting instrumentation. - Content plan: immersive model viewport, focused inspection rail, editing and gaussian workflows, export and publish handoff. - Interaction thesis: smooth orbit-first camera motion, intentional inspection overlays, and progressive reveal from low-latency preview to full-fidelity asset. ## Verification ```bash NX_DAEMON=false pnpm nx test @isis/3d-browser NX_DAEMON=false pnpm nx lint @isis/3d-browser NX_DAEMON=false pnpm nx build @isis/3d-browser pnpm tsc -p libs/isis/3d-browser/tsconfig.json --noEmit ```