rlbook.ai
Chapter 13
📝Draft

The Bellman Equations

The recursive equations that make RL possible

Prerequisites:

What You'll Learn

  • Derive the Bellman expectation equation for VπV^\pi and QπQ^\pi
  • Derive the Bellman optimality equations for V∗V^* and Q∗Q^*
  • Explain what “bootstrapping” means and why it enables efficient learning
  • Understand why Bellman equations are the foundation of all RL algorithms

Richard Bellman discovered something remarkable in the 1950s: the value of a state depends on the values of states you can reach from it. This recursive insight—that you can break down a complex problem into simpler subproblems—is the foundation of everything in reinforcement learning.

You can watch that insight at work right now. Press Play: each sweep applies the Bellman backup you’re about to learn, once to every cell.

Value Iteration Visualization

Watch value iteration solve the GridWorld problem step by step.

0.0
S
0.0
0.0
0.0
0.0
🧱
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
🎯
Click "Step" or "Play" to start value iteration
0
Iteration
0.0000
Max Delta
0.90
Gamma
🎯Goal (+10)
🧱Obstacle
High Value
Low Value
How it works: Value iteration repeatedly applies the Bellman equation V(s) = max_a [R(s,a) + γ V(s')] until values converge. Each cell shows its estimated value, and arrows show the optimal policy. Higher gamma values make the agent care more about future rewards. More negative step rewards encourage shorter paths.

The Recursive Nature of Value

Consider standing at a crossroads. How valuable is your current position? It depends on:

  1. What reward you get right now (maybe there’s a rest stop here)
  2. Where you can go from here (the available roads)
  3. How valuable those destinations are (what awaits down each path)

But wait—the value of those destinations depends on their successors. And those depend on further successors. It seems like an infinite regress!

Bellman’s insight was that this recursion is actually a feature, not a bug. The value of any state can be expressed in terms of immediate rewards plus the values of neighboring states. This creates a system of equations that we can solve.

📖Bellman Equations

Recursive relationships that express the value of a state (or state-action pair) in terms of immediate rewards and the values of successor states. They form a system of equations that value functions must satisfy.

The Core Insight: Value Flows Backward

Think about how you might evaluate different starting positions in a maze:

📌A Simple Maze

Consider a 3-state path: Start, Middle, Goal

Start
→
Mid
→
+10

With γ=0.9\gamma = 0.9 and the +10 earned on the transition from Mid into the terminal Goal:

  • Goal value: V(Goal)=0V(\text{Goal}) = 0 (terminal: nothing happens after arriving)
  • Middle value: V(Mid)=10+γ⋅V(Goal)=10+0.9⋅0=10V(\text{Mid}) = 10 + \gamma \cdot V(\text{Goal}) = 10 + 0.9 \cdot 0 = 10
  • Start value: V(Start)=0+γ⋅V(Mid)=0+0.9⋅10=9V(\text{Start}) = 0 + \gamma \cdot V(\text{Mid}) = 0 + 0.9 \cdot 10 = 9

Value flows backward from rewards through the state space.

This backward flow is the essence of the Bellman equations. Each state’s value depends on its successors’ values, which depend on their successors’ values, all the way to the rewards.

Chapter Overview

Bellman Expectation Equations
How values relate under a specific policy
Bellman Optimality Equations
The equations that define optimal behavior
Why Bellman Matters
Bootstrapping and the foundation of RL

The Two Types of Bellman Equations

There are two families of Bellman equations, and understanding the difference is crucial:

Expectation Equations
"If I follow this particular strategy, how good is each state?"
  • Describe values under a specific policy π
  • AVERAGE over actions, weighted by the policy
Optimality Equations
"If I act optimally from now on, how good is each state?"
  • Describe the best possible values
  • Take the MAX over actions instead of averaging
∑Mathematical Details

Bellman expectation equation (average over the policy’s actions):

Vπ(s)=∑aπ(a∣s)[R(s,a)+γ∑s′P(s′∣s,a)Vπ(s′)]V^\pi(s) = \sum_{a} \pi(a|s) \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V^\pi(s') \right]

Bellman optimality equation (maximum over actions):

V∗(s)=max⁡a[R(s,a)+γ∑s′P(s′∣s,a)V∗(s′)]V^*(s) = \max_a \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V^*(s') \right]

ℹ️Note

The shift from averaging over the policy’s actions to maximizing over actions is subtle but profound. It means the optimality equations don’t require knowing the optimal policy—the max⁡\max operator embeds the optimal action choice directly into the equation.

Why This Chapter Matters

The Bellman equations are not just theoretical curiosities—they’re the foundation of every major RL algorithm:

Dynamic Programming
Solves Bellman equations exactly when the model is known
Monte Carlo
Estimates values by sampling complete returns
Temporal Difference
Approximates Bellman with single-step sampled updates
Q-Learning
Learns optimal values via sampled Bellman optimality updates
💡Tip

If you understand the Bellman equations deeply, you understand the core of reinforcement learning. Everything else is about how to solve or approximate these equations under different conditions—when you know the model, when you don’t, when states are discrete, when they’re continuous.

The Journey Ahead

Expectation
derive values under a policy
Optimality
swap the average for a max
Bootstrapping
update estimates with estimates
Algorithms
policy evaluation and beyond

The payoff comes immediately in the next section of the book: Policy Evaluation turns the Bellman expectation equation directly into an algorithm.

∑Mathematical Details

By the end of this chapter, you’ll be comfortable with all four Bellman equations:

For state values:

  • Expectation: Vπ(s)=∑aπ(a∣s)[R(s,a)+γ∑s′P(s′∣s,a)Vπ(s′)]V^\pi(s) = \sum_{a} \pi(a|s) \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V^\pi(s') \right]
  • Optimality: V∗(s)=max⁡a[R(s,a)+γ∑s′P(s′∣s,a)V∗(s′)]V^*(s) = \max_a \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V^*(s') \right]

For action values:

  • Expectation: Qπ(s,a)=R(s,a)+γ∑s′P(s′∣s,a)∑a′π(a′∣s′)Qπ(s′,a′)Q^\pi(s,a) = R(s,a) + \gamma \sum_{s'} P(s'|s,a) \sum_{a'} \pi(a'|s') Q^\pi(s',a')
  • Optimality: Q∗(s,a)=R(s,a)+γ∑s′P(s′∣s,a)max⁡a′Q∗(s′,a′)Q^*(s,a) = R(s,a) + \gamma \sum_{s'} P(s'|s,a) \max_{a'} Q^*(s',a')

Summary

  1. Values are recursive
    The value of a state is the immediate reward plus the discounted value of its successors.
  2. Expectation averages, optimality maximizes
    One family describes a given policy; the other describes the best possible behavior — the only difference is average vs. max over actions.
  3. Value flows backward from rewards
    Each backup pushes reward information one step further from its source, as you saw in the demo.
  4. The max operator embeds the optimal policy
    You can characterize optimal values without knowing the optimal policy in advance.
  5. Every RL algorithm is Bellman in disguise
    DP solves the equations exactly; TD and Q-learning sample them.
Check your understanding
1. The value function is defined as an expectation over infinitely many possible futures. What turns this definition into an equation we can actually solve?
2. What is the single structural difference between the Bellman expectation and Bellman optimality equations?
3. Why can you initialize every value to zero (or random numbers) and still converge to the true values by repeatedly applying Bellman backups?
4. What does "bootstrapping" mean in the context of RL?
5. The Bellman expectation equation can be solved by matrix inversion, but the optimality equation cannot. Why?
 Next ChapterPolicy Evaluation→