Why Bellman Matters | The Bellman Equations

The Bellman equations aren’t just mathematics—they’re the engine that powers all of reinforcement learning. Understanding why they matter is as important as knowing what they say.

The Power of Recursion

The Bellman equations reveal a remarkable property of value functions: values are self-consistent.

📖Self-Consistency of Values

If $V$ is the true value function for a policy $\pi$ , then the value at any state is exactly what you’d compute from its neighbors’ values: the expected immediate reward plus the discounted, probability-weighted value of the successor states.

∑Mathematical Details

Formally, for every state $s$ :

$V(s) = \sum_a \pi(a|s) \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V(s') \right]$

This self-consistency has a remarkable implication: if we find a value function that satisfies the Bellman equation at every state, we’ve found the true value function.

It’s like a giant Sudoku puzzle. Each cell’s value is constrained by its neighbors. If all constraints are satisfied simultaneously, the solution is correct.

Bootstrapping: The Key Insight

📖Bootstrapping

Using an estimate of the value function to update that same estimate. The Bellman equations tell us that values depend on other values—so we can use current estimates to improve our estimates.

The name comes from the phrase “pulling yourself up by your bootstraps.” It seems impossible—how can you improve estimates using those same (potentially wrong) estimates?

Here’s the magic: even if your initial estimates are completely wrong, applying the Bellman equation consistently will drive them toward the truth. Why? Because each update:

Uses the actual reward from the environment (ground truth)
Makes the value locally consistent with neighbors
When all values are locally consistent, they’re globally correct

📌Bootstrapping in Action

Consider three states in a line: A, B, C. C is terminal, and the transition from B into C pays +10 (all other rewards are 0, $\gamma = 0.9$ ). Terminal states have value 0.

Initial estimates:

$V(A) = 0$ , $V(B) = 0$ , $V(C) = 0$

After one Bellman backup at B:

$V(B) = 10 + 0.9 \times V(C) = 10 + 0.9 \times 0 = 10$

After another backup at A:

$V(A) = 0 + 0.9 \times V(B) = 0 + 0.9 \times 10 = 9$

9.0

→

+10
→

C (terminal)

Value “propagated” from the reward on the B→C transition backward through the chain. Each update used the current estimate of the next state—that’s bootstrapping.

Why Bootstrapping Works

∑Mathematical Details

Define the Bellman operator $\mathcal{T}$ for a policy $\pi$ :

$(\mathcal{T}^\pi V)(s) = \sum_a \pi(a|s) \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V(s') \right]$

Then for any two value functions $V_1$ and $V_2$ :

$\|\mathcal{T}^\pi V_1 - \mathcal{T}^\pi V_2\|_\infty \leq \gamma \|V_1 - V_2\|_\infty$

Since $\gamma < 1$ , each application of the operator shrinks the distance to at most $\gamma$ times its previous value. After $k$ iterations:

$\|V_k - V^\pi\|_\infty \leq \gamma^k \|V_0 - V^\pi\|_\infty \rightarrow 0$

Think of it like this: no matter how wrong your initial guess is, each iteration cuts the remaining error to at most $\gamma$ times what it was — you close at least a $(1-\gamma)$ fraction of the gap every step. After enough iterations, you’re arbitrarily close.

This is why you can initialize values to zero (or random numbers) and still converge. The contraction property guarantees it.

💡Tip

The contraction property explains why we need $\gamma < 1$ for infinite-horizon problems. If $\gamma = 1$ , the operator might not contract, and convergence isn’t guaranteed (though special cases like episodic tasks can still work).

Local Updates, Global Solutions

One of the most remarkable aspects of Bellman equations is that local updates lead to global solutions. Each Bellman backup only looks at immediate neighbors—the current state, its actions, its one-step successors, and one-step rewards—yet repeated local updates produce globally correct values, because value information propagates through the state space one backup at a time.

You can watch this propagation live. Press Step to apply one sweep of backups at a time and see the reward’s influence spread outward from the goal:

Value Iteration Visualization

Watch value iteration solve the GridWorld problem step by step.

Gamma (γ):0.90Step Reward:Goal Reward:Speed:SlowFast

0.0

🧱

0.0

🎯

Click "Step" or "Play" to start value iteration

Iteration

0.0000

Max Delta

0.90

Gamma

🎯Goal (+10)

🧱Obstacle

High Value

Low Value

How it works: Value iteration repeatedly applies the Bellman equation V(s) = max_a [R(s,a) + γ V(s')] until values converge. Each cell shows its estimated value, and arrows show the optimal policy. Higher gamma values make the agent care more about future rewards. More negative step rewards encourage shorter paths.

📌Value Propagation in GridWorld

Consider a 4x4 GridWorld with a goal in the corner. Entering the goal pays +10; every other transition pays 0; the goal is terminal (value 0); $\gamma = 0.9$ . Moves that would leave the grid keep the agent in place.

+10

Initial values (all 0; the +10 marks the reward for entering the goal)

After several iterations of Bellman backups (with $\gamma = 0.9$ ):

8.1

9.0

10.0

7.3

8.1

9.0

10.0

6.6

7.3

8.1

9.0

5.9

6.6

7.3

8.1

Converged values: 10.0 one step from the goal, shrinking by a factor of 0.9 per extra step

Values form a “gradient” that points toward the goal. Following the gradient (moving to higher-valued neighbors) leads to the optimal path.

The Foundation of RL Algorithms

Every major RL algorithm relates to the Bellman equations in some way. Understanding this connection helps you see the unity beneath the diversity of methods.

Dynamic Programming

Solves the Bellman equations exactly when the MDP model is known.

Policy evaluation applies the expectation backup
Value iteration applies the optimality backup

Monte Carlo

Estimates values by averaging complete sampled returns — no bootstrapping.

Uses true returns, so no bias
But returns vary widely, so high variance

Temporal Difference

Samples the Bellman equation one step at a time — this IS bootstrapping.

Updates from actual reward plus estimated next value
Some bias, but much lower variance

Q-Learning

A sampled version of the Bellman optimality equation for Q.

The max inside the update targets optimal values
Learns them regardless of the behavior policy

∑Mathematical Details

The update rules, side by side:

Policy evaluation (DP, expectation backup):

$V_{k+1}(s) = \sum_a \pi(a|s) \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V_k(s') \right]$

Value iteration (DP, optimality backup):

$V_{k+1}(s) = \max_a \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V_k(s') \right]$

Monte Carlo (no bootstrapping; $G_t$ is the complete observed return):

$V(s) \leftarrow V(s) + \alpha (G_t - V(s))$

TD learning (bootstraps on the estimate $V(s')$ ):

$V(s) \leftarrow V(s) + \alpha \left( r + \gamma V(s') - V(s) \right)$

Q-learning (sampled optimality backup):

$Q(s,a) \leftarrow Q(s,a) + \alpha \left( r + \gamma \max_{a'} Q(s',a') - Q(s,a) \right)$

ℹ️Note

Notice the pattern: DP uses the full Bellman equation, TD/Q-learning sample it. This is the fundamental trade-off in RL: exact but expensive (DP) vs. approximate but efficient (TD).

Bellman Equations in Deep RL

Even in deep reinforcement learning with neural networks, Bellman equations remain central.

📌DQN: Deep Q-Networks

DQN approximates $Q^*$ with a neural network $Q_\theta(s, a)$ and trains it to satisfy the Bellman optimality equation: the loss penalizes the squared gap between the network’s Q-value and the Bellman target (reward plus discounted best next Q-value). The target is still the Bellman backup—just computed with a neural network instead of a table.

∑Mathematical Details

$\text{Loss} = \mathbb{E}\left[ \left( r + \gamma \max_{a'} Q_{\theta^-}(s', a') - Q_\theta(s, a) \right)^2 \right]$

The network learns to make the Bellman equation hold (approximately) across all state-action pairs.

</>Implementation

Here’s the Bellman-based loss function used in DQN:

def compute_dqn_loss(q_network, target_network, batch, gamma=0.99):
    """
    Compute the DQN loss based on Bellman optimality equation.

    Args:
        q_network: The Q-network being trained
        target_network: Frozen copy for stable targets
        batch: (states, actions, rewards, next_states, dones)
        gamma: Discount factor

    Returns:
        Mean squared Bellman error
    """
    states, actions, rewards, next_states, dones = batch

    # Current Q-values for taken actions
    q_values = q_network(states)
    q_values = q_values.gather(1, actions.unsqueeze(1)).squeeze(1)

    # Target: Bellman optimality equation
    with torch.no_grad():
        next_q_values = target_network(next_states)
        max_next_q = next_q_values.max(dim=1)[0]

        # If done, no future value
        target = rewards + gamma * max_next_q * (1 - dones)

    # Loss: How far are we from satisfying Bellman?
    loss = torch.nn.functional.mse_loss(q_values, target)

    return loss

Why This Matters for You

Understanding the Bellman equations deeply gives you:

Intuition for debugging: When an RL algorithm fails, ask: “Is the Bellman equation being satisfied? Where is it breaking down?”
Algorithm design: Many RL innovations are creative ways to approximate or modify Bellman updates (n-step returns, eligibility traces, distributional RL).
Unified perspective: All RL algorithms are trying to solve or approximate the Bellman equations. Once you see this, the field becomes more coherent.

💡Tip

When learning a new RL algorithm, always ask: “How does this relate to the Bellman equations?” You’ll find the answer illuminating.

Common Misconceptions

Before we move on, let’s address some common points of confusion:

Looking Ahead

The Bellman equations tell us what optimal values look like, but not how to find them efficiently. The chapters ahead tackle that challenge, starting with Policy Evaluation—which turns the expectation backup you’ve just learned directly into an algorithm—and continuing through Monte Carlo estimation, TD learning, and model-free Q-learning.

Summary

The Bellman equations are the mathematical heart of reinforcement learning:

Self-consistency: Values must satisfy the Bellman equation at every state
Bootstrapping: We can use estimates to improve estimates, converging to truth
Contraction: The Bellman operator shrinks errors, guaranteeing convergence
Local to global: Each update only looks at neighbors, yet globally optimal values emerge
Universal foundation: Every RL algorithm is solving or approximating these equations

ℹ️Note

Richard Bellman’s insight in the 1950s was that complex sequential decision problems have recursive structure. This “principle of optimality” underlies not just RL, but optimal control, operations research, and many other fields. In RL, it’s the foundation everything else builds on.

Understanding why Bellman matters gives you more than theoretical knowledge—it gives you the intuition to understand, debug, and design RL algorithms. When something goes wrong, you can trace it back to the Bellman equations. When designing something new, you can think about how it relates to these fundamental principles.