rlbook.ai
Chapter 303
📝Draft

Offline RL

Learning from logged data without environment interaction

Prerequisites:

What You'll Learn

  • Explain why offline RL is important for real-world applications
  • Describe the distribution shift problem
  • Implement Conservative Q-Learning (CQL)
  • Explain behavior cloning and its limitations
  • Understand the tradeoff between conservatism and optimality
  • Identify when offline RL is appropriate vs online RL

What if you can’t explore? In healthcare, you can’t experiment on patients to learn a treatment policy. In autonomous driving, you can’t crash cars to learn safe behavior. In industrial control, you can’t risk damaging expensive equipment.

But you have years of logged data from doctors, human drivers, and plant operators. Can you learn good policies from this fixed dataset, without any new interaction?

Offline RL is learning to drive from dashcam footage. You watch thousands of hours of driving videos: what human drivers did, what happened as a result. But you never actually get behind the wheel during training. Can you learn to drive well?

The answer is: yes, but it’s tricky. The challenge is that your policy might want to do something the humans never did—and you have no data about what happens then.

Chapter Overview

In this chapter:
The Offline Setting
Why offline RL matters and when to use it: learning from fixed datasets without environment interaction
Distribution Shift
The core challenge: why standard RL algorithms fail on offline data and how Q-values become unreliable
Conservative Methods
Solutions like CQL that stay close to the data by penalizing out-of-distribution actions

The Big Picture

📖Offline RL

Reinforcement learning from a fixed dataset of previously collected experience, with no ability to interact with the environment during training. Also called “batch RL” or “data-driven RL.”

Offline RL learns from a fixed dataset without environment interaction. The core challenge is distribution shift: the learned policy might choose actions never seen in the data, and we have no way to know if those actions are good or catastrophic. Conservative methods explicitly discourage out-of-distribution actions.

The key difference from online RL:

Online RL: Try something, see what happens, learn from it. If you’re uncertain about an action, you can explore and find out.

Offline RL: You only have the data you have. If you’re uncertain about an action not in the data, you can’t explore—you must either trust your extrapolation or avoid that action entirely.

This fundamental constraint changes everything about how we approach the problem.

Why Offline RL Matters

Real-World Applications
  • Healthcare: treatment policies from medical records
  • Autonomous driving: learning from human demonstrations
  • Robotics: using collected teleoperation data
  • Industrial control: optimizing from historical operations
Key Benefits
  • Safety: no risky exploration during training
  • Leverage existing data: use what you already have
  • Reproducibility: same data gives same training
  • Cost-effective: no need for expensive simulators
💡Start Here

New to offline RL? Begin with The Offline Setting to understand when and why offline RL is needed, then continue through the sections in order.

Connection to RLHF

ℹ️Looking Ahead

Offline RL ideas echo in how modern language models are trained, though the connection is partial. In RLHF (Reinforcement Learning from Human Feedback), the human preference data used to train the reward model is collected offline — but the subsequent PPO stage is online RL against that learned reward model, with the policy generating fresh samples. The offline-data mindset from this chapter — distribution shift, exploiting a fixed dataset’s blind spots — still lays useful groundwork for the next chapter.

Check Your Understanding

Check your understanding
1. What defines the offline RL setting?
2. Why does running vanilla Q-learning on offline data fail?
3. How does Conservative Q-Learning (CQL) address distribution shift?
4. What is behavior cloning's fundamental ceiling?
5. Which statement about the RLHF connection is accurate?
  1. A fixed dataset, and nothing else
    Offline RL must produce a policy from logged experience—no exploration, no corrective feedback from the environment.
  2. Distribution shift is the central enemy
    The learned policy wants actions the data never covered; their Q-values are unchecked extrapolations that the max operator loves to select.
  3. Conservatism is the standard remedy
    CQL penalizes out-of-distribution actions and provably under-estimates value in expectation; BCQ constrains the policy to data-like actions.
  4. Coverage beats volume
    What the behavior policy explored determines what you can learn—a narrow expert dataset supports imitation but punishes deviation.
  5. The ideas travel
    Safety-critical domains (healthcare, driving, industrial control) and parts of LLM training pipelines all inherit offline RL's distribution-shift discipline.

Primary Sources

  • Levine, Kumar, Tucker & Fu (2020), “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems” — the standard survey of the field. arXiv:2005.01643
  • Fujimoto, Meger & Precup (2019), “Off-Policy Deep Reinforcement Learning without Exploration,” ICML 2019 — BCQ, and the diagnosis of extrapolation error. arXiv:1812.02900
  • Kumar, Zhou, Tucker & Levine (2020), “Conservative Q-Learning for Offline Reinforcement Learning,” NeurIPS 2020 — CQL and its lower-bound guarantee. arXiv:2006.04779
  • Chen et al. (2021), “Decision Transformer: Reinforcement Learning via Sequence Modeling,” NeurIPS 2021 — offline RL as conditional sequence modeling. arXiv:2106.01345
Next ChapterRL for Language Models→