DreamerV3-class model-based reinforcement learning (Phase 176.3), implemented in PyTorch with CPU-runnable forward and backward passes verified by overfit tests.
Components#
- RSSM world model (
rssm.py,world_model.py) — deterministic GRU branch- 32×32 categorical stochastic branch with a straight-through sampler, prior / posterior heads, KL balancing (α=0.8), free bits (1.0 nat), unimix 1 %, symlog observation reconstruction, and twohot symexp reward + Bernoulli continue heads. (176.3.1.1)
- Actor-critic in imagination (
actor_critic.py) — backward λ-returns (λ=0.95), percentile return normalisation, annealed entropy schedule (3e-4 → 1e-4), twohot value head with a slow EMA target, and gradient clipping at norm 100. Discrete and tanh-Normal continuous actors. (176.3.1.2) - Offline mode (
offline.py) — trains the world model and imagination actor-critic from a fixed dataset of logged flywheel trajectories with a behaviour-regularisation term, no live environment. (176.3.2.2) - Hierarchical Director (
hierarchical.py) — a goal autoencoder (discrete goal codebook), a manager policy acting every K steps, and a goal-conditioned worker with cosine-gain intrinsic reward. (176.3.2.3) - DayDreamer loop (
daydreamer.py) — bounded episode-safe robot replay and interleaved environment/training cadence with action repeat. This implements the on-robot algorithm seam; the one-hour quadruped result remains external. - VLM conditional prior (
vlm_prior.py) — injected PaliGemma goal encoder and gated language/vision-conditioned RSSM categorical prior for multi-task Dreamer. The production checkpoint remains an external asset.
Not included (require GPU + full datasets — left unchecked in TODOS)#
176.3.1.3 (DreamerV3-XL 200M on Crafter/Minecraft/Atari-200M), the physical
quadruped training/terrain-transfer outcome in 176.3.1.4, and the trained
PaliGemma checkpoint evaluation in 176.3.2.1 need accelerated training,
hardware, and external assets.
Tests#
text
cd libs/nous/dreamer && python3 -m pytest tests/ -q
The suite verifies the symlog/twohot algebra against closed forms, the RSSM KL/free-bits/straight-through behaviour, and — crucially — that the world model overfits a fixed sequence, the actor-critic solves an imagined bandit, the offline trainer reduces world loss, and the goal autoencoder reconstructs compressible features.