Eve library

nous-dreamer

Other

Revision 1

On this page

DreamerV3-class model-based reinforcement learning (Phase 176.3), implemented in PyTorch with CPU-runnable forward and backward passes verified by overfit tests.

Components#

  • RSSM world model (rssm.py, world_model.py) — deterministic GRU branch
    • 32×32 categorical stochastic branch with a straight-through sampler, prior / posterior heads, KL balancing (α=0.8), free bits (1.0 nat), unimix 1 %, symlog observation reconstruction, and twohot symexp reward + Bernoulli continue heads. (176.3.1.1)
  • Actor-critic in imagination (actor_critic.py) — backward λ-returns (λ=0.95), percentile return normalisation, annealed entropy schedule (3e-4 → 1e-4), twohot value head with a slow EMA target, and gradient clipping at norm 100. Discrete and tanh-Normal continuous actors. (176.3.1.2)
  • Offline mode (offline.py) — trains the world model and imagination actor-critic from a fixed dataset of logged flywheel trajectories with a behaviour-regularisation term, no live environment. (176.3.2.2)
  • Hierarchical Director (hierarchical.py) — a goal autoencoder (discrete goal codebook), a manager policy acting every K steps, and a goal-conditioned worker with cosine-gain intrinsic reward. (176.3.2.3)
  • DayDreamer loop (daydreamer.py) — bounded episode-safe robot replay and interleaved environment/training cadence with action repeat. This implements the on-robot algorithm seam; the one-hour quadruped result remains external.
  • VLM conditional prior (vlm_prior.py) — injected PaliGemma goal encoder and gated language/vision-conditioned RSSM categorical prior for multi-task Dreamer. The production checkpoint remains an external asset.

Not included (require GPU + full datasets — left unchecked in TODOS)#

176.3.1.3 (DreamerV3-XL 200M on Crafter/Minecraft/Atari-200M), the physical quadruped training/terrain-transfer outcome in 176.3.1.4, and the trained PaliGemma checkpoint evaluation in 176.3.2.1 need accelerated training, hardware, and external assets.

Tests#

text
cd libs/nous/dreamer && python3 -m pytest tests/ -q

The suite verifies the symlog/twohot algebra against closed forms, the RSSM KL/free-bits/straight-through behaviour, and — crucially — that the world model overfits a fixed sequence, the actor-critic solves an imagined bandit, the offline trainer reduces world loss, and the goal autoencoder reconstructs compressible features.