rlbook.ai
Chapter 301
📝Draft

Model-Based RL

Learning world models for sample-efficient planning

Prerequisites:

What You'll Learn

  • Distinguish between model-free and model-based RL
  • Explain the sample efficiency advantage of model-based methods
  • Implement the Dyna architecture
  • Describe how to learn environment models
  • Explain the model bias problem and how to mitigate it
  • Describe modern model-based methods including MuZero

Every model-free method we’ve seen learns by trial and error in the real world. But what if the agent could imagine experiences? What if it had a model of how the world works and could plan ahead, like a chess player thinking several moves ahead?

This is the fundamental insight behind model-based reinforcement learning: if you understand how the world works, you can simulate experiences in your head and learn from them without ever actually interacting with the environment.

Think about how you plan a road trip. You don’t need to actually drive every possible route to know which is fastest. Instead, you have a mental model of the road network, traffic patterns, and driving times. You use this model to plan, evaluating routes in your imagination before committing to one.

Model-based RL gives agents this same capability: learn how the world works, then use that knowledge to plan efficiently.

Chapter Overview

In this chapter:
Learning World Models
What is a model, how to learn one from experience, and the fundamental tradeoff with model-free methods
Planning with Learned Models
Using imagination for better decisions: how models enable planning and sample efficiency
The Dyna Architecture
Combining real and simulated experience in a unified framework for sample-efficient learning
MuZero and Beyond
Modern approaches that learn abstract models optimized for planning, without explicit state prediction

The Big Picture

📖Environment Model

A learned approximation of the environment’s dynamics: the transition function P(s′∣s,a)P(s'|s,a) that predicts next states, and the reward function R(s,a)R(s,a) that predicts rewards. With a model, agents can “imagine” experiences without actually interacting with the environment.

These are the same two ingredients that define an MDP in Introduction to MDPs—the difference is that back then we assumed they were given, and the dynamic programming methods of Policy Improvement planned with them exactly. Model-based RL earns those ingredients from data.

Model-based RL learns a model of the environment (how states transition, what rewards occur) and uses it to plan or generate synthetic experience. This can be far more sample-efficient than model-free learning, but introduces the challenge of model errors.

Consider the difference:

  • Model-free: Learn solely from real interactions. Each environment step is precious and used once.
  • Model-based: Learn a model from real interactions, then use that model to generate unlimited simulated experience.

It’s like the difference between learning to cook by only making real dishes (expensive, slow) versus first understanding the principles of cooking (what flavors combine well, how heat affects food) and then practicing in your head before touching ingredients.

When to Use Model-Based RL

Model-Based is Great When:
  • Real interactions are expensive or slow
  • Safety is critical (robotics, healthcare)
  • The environment dynamics are relatively simple
  • You need sample efficiency
Model-Free May Be Better When:
  • Environment dynamics are very complex
  • You have abundant simulation access
  • Model errors would compound badly
  • Compute is more expensive than samples
💡Start Here

New to model-based RL? Begin with Learning World Models to understand what models are and how they’re learned, then continue through the sections in order.

Check Your Understanding

Check your understanding
1. What actually distinguishes a model-based method from a model-free one?
2. In Dyna-Q, what happens during the planning steps between real environment interactions?
3. Why can training a policy purely on model rollouts backfire even when the model is mostly accurate?
4. What does MuZero's learned model predict?
5. Model-based methods are usually more sample-efficient. What do they pay for it?
  1. Learn the world, then plan in it
    A model of P(s'|s,a) and R(s,a) lets the agent generate imagined experience instead of paying for every real interaction.
  2. Dyna unifies learning and planning
    Real and simulated transitions feed the identical Q-learning update—one loop, arbitrarily amplified.
  3. Sample efficiency has a price
    Planning costs compute, and everything now depends on model quality.
  4. Model errors compound
    Small per-step inaccuracies snowball over long rollouts, and policy optimization actively exploits the model's blind spots.
  5. Modern models predict what matters
    MuZero and its successors model rewards, values, and policies in latent space—decision-relevant structure, not pixels.

Primary Sources

  • Sutton (1991), “Dyna, an Integrated Architecture for Learning, Planning, and Reacting,” SIGART Bulletin 2(4) — the original Dyna paper
  • Silver et al. (2016), “Mastering the game of Go with deep neural networks and tree search,” Nature 529 — AlphaGo. Nature page
  • Silver et al. (2017), “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm” — AlphaZero. arXiv:1712.01815
  • Schrittwieser et al. (2020), “Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model,” Nature 588 — MuZero. arXiv:1911.08265
  • Ha & Schmidhuber (2018), “World Models” — learning to act inside a learned latent simulator. arXiv:1803.10122
  • Hafner et al. (2020), “Dream to Control: Learning Behaviors by Latent Imagination,” ICLR 2020 — Dreamer. arXiv:1912.01603
Next ChapterMulti-Agent RL→