Chapter 12

📝Draft

Value Functions

Measuring how good states and actions are

Prerequisites:

Introduction to MDPs

What You'll Learn

Define state-value and action-value functions
Compute values for simple MDPs by hand
Explain the relationship between V(s) and Q(s,a)
Define optimal value functions V* and Q*
Explain why knowing optimal values lets us derive optimal policies

MDPs gave us a language for describing sequential decision problems. Now we need a way to measure success within them. How good is it to be in a particular state? How good is a particular action? Value functions answer these questions—and they’re the key to finding optimal behavior.

Before any definitions, watch values in action. Press Play and watch numbers propagate through the grid—this chapter explains what those numbers mean.

Value Iteration Visualization

Watch value iteration solve the GridWorld problem step by step.

Gamma (γ):0.90Step Reward:Goal Reward:Speed:SlowFast

0.0

🧱

0.0

🎯

Click "Step" or "Play" to start value iteration

Iteration

0.0000

Max Delta

0.90

Gamma

🎯Goal (+10)

🧱Obstacle

High Value

Low Value

How it works: Value iteration repeatedly applies the Bellman equation V(s) = max_a [R(s,a) + γ V(s')] until values converge. Each cell shows its estimated value, and arrows show the optimal policy. Higher gamma values make the agent care more about future rewards. More negative step rewards encourage shorter paths.

The Central Question

A robot navigating a building, an agent playing a game, an algorithm making trades—at any moment, each faces the same question: “How am I doing?” Answering it requires thinking about the future. A position that looks good now might lead to disaster later; a sacrifice today might pay off tomorrow.

Imagine playing chess. You look at the board and think: “Am I winning or losing?”

That intuitive assessment—collapsing all future possibilities into a single judgment—is exactly what value functions do. They tell you how “good” a position is, accounting for everything that might happen next.

📖Value Function

A function that maps states (or state-action pairs) to the expected cumulative reward an agent can achieve from that point forward, following a particular policy.

Value functions compress the infinite complexity of possible futures into a single number. If you know $V^*(s)$ , you know everything you need about the long-term prospects of being in state $s$ . This compression is what makes planning tractable.

Two Types of Value Functions

State value — V(s)

How good is it to BE in this state?

Expected return starting from state s
Follows policy π from there on

Action value — Q(s,a)

How good is it to TAKE this action here?

Expected return after taking action a in state s
Then follows policy π

The key difference: V-values tell you where to be, Q-values tell you what to do. This makes Q-values more directly useful for decision-making.

Values Depend on Policy

A crucial insight: the value of a state depends on the policy. The same state can have very different values under different policies.

📌Policy Matters

Take the 4x4 GridWorld studied in this chapter (each step costs $-1$ , entering the goal pays +10, $\gamma = 0.9$ ) and consider a cell 5 steps from the goal:

Under an optimal policy (always moves toward goal):

Value is $\approx +2.47$ (five step costs, then the discounted +10)

Under a random policy (wanders aimlessly):

Value is $\approx -8.6$ (step penalties pile up while wandering)

Same state, same MDP, vastly different values. The policy determines the value.

ℹ️Note

Value functions are predictions. They predict the expected cumulative reward. Good predictions enable good decisions: if you know the value of every state you could end up in, you can evaluate any policy and make intelligent choices.

Chapter Overview

State Value Functions

The formal definition of V(s), how values depend on policy, and how to compute them

Action Value Functions

The Q-function, why it is more useful for control, and the V-Q relationship

Optimal Value Functions

The best possible values V* and Q*, and how they lead to optimal policies

The Key Equations

In plain English: V averages the returns you get from a state; Q averages the returns you get after committing to one action first; the optimal versions ask for the best achievable numbers; and once you have $Q^*$ , acting optimally is just picking the highest-rated action.

∑Mathematical Details

State-value function: $V^\pi(s) = \mathbb{E}_\pi[G_t | S_t = s] = \mathbb{E}_\pi\left[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} \bigg| S_t = s\right]$

Action-value function: $Q^\pi(s, a) = \mathbb{E}_\pi[G_t | S_t = s, A_t = a]$

V-Q relationship: $V^\pi(s) = \sum_{a \in \mathcal{A}} \pi(a|s) Q^\pi(s, a)$

Optimal values: $V^*(s) = \max_\pi V^\pi(s) \qquad Q^*(s, a) = \max_\pi Q^\pi(s, a)$

Optimal policy from Q*: $\pi^*(s) = \arg\max_a Q^*(s, a)$

Visual Preview: The Value Landscape

In a GridWorld, value functions create a “landscape” that reveals the structure of the problem:

📌GridWorld Values

Consider a 4x4 grid with a goal in the corner ( $-1$ per step, +10 for entering the goal, $\gamma = 0.9$ ; the goal is terminal with value 0). Under an optimal policy:

Values (higher = better):
 _____ _____ _____ _____
|     |     |     |     |
| 1.2 | 2.5 | 3.9 | 5.4 |
|_____|_____|_____|_____|
|     |     |     |     |
| 2.5 |  X  | 5.4 | 7.1 |
|_____|_____|_____|_____|
|     |     |     |     |
| 3.9 | 5.4 | 7.1 | 9.0 |
|_____|_____|_____|_____|
|     |     |     |     |
| 5.4 | 7.1 | 9.0 |  G  |
|_____|_____|_____|_____|
                    Goal

The values increase as you approach the goal, forming a gradient that “points” toward the reward. Following this gradient is essentially what an optimal policy does—and it’s the same gradient you saw emerge in the demo above.

What Comes Next

After understanding value functions, the natural question is: How do we compute them? The answer lies in the Bellman equations: the value of a state depends on the values of its successor states. This recursion enables efficient computation through dynamic programming—you’ll see it put to work in Policy Evaluation.

MDPs

define the problem

Value functions

measure quality

Bellman equations

enable computation

Algorithms

solve for values

Summary

V(s) rates states, Q(s,a) rates actions
Both measure expected return under a policy π; Q commits to one action first.
Values always depend on the policy
The same state can be worth +2.47 under a good policy and -8.6 under a bad one.
V and Q are two views of the same quantity
V(s) is the policy-weighted average of the Q-values at s.
Optimal values are the ceiling
V* and Q* describe the best any policy can possibly achieve.
Q* makes acting optimally trivial
Just pick the action with the highest Q*-value in each state — no model needed.

Check your understanding

1. What is the difference between the return and the value of a state?

They are the same quantity measured at different time stepsThe return is the discounted reward sum of one particular trajectory; the value is the expected return over all possible trajectoriesThe value includes the immediate reward, while the return does notThe return applies only to episodic tasks, the value only to continuing tasks

2. In the GridWorld above, entering the goal pays +10 — yet the goal cell itself shows a value of 0. Why?

The discount factor shrinks the reward to zero at the goalThe +10 is earned on the transition INTO the goal; a terminal state has no future rewards, so its value is 0The solver did not converge at that cellTerminal values are defined as the negative of the step cost

3. You know V*(s) for every state, but nothing else about the MDP. Can you act optimally?

Yes — always move toward the neighboring state you happen to reachOnly if you also know the transition dynamics P(s'|s,a), so you can do a one-step lookaheadYes — V* alone directly determines the optimal actionNo — optimal behavior requires learning from scratch regardless of what you know

4. The same state has value +2.47 in one computation and -8.6 in another, with identical rewards and discount. What explains this?

One of the two computations must contain an arithmetic errorThe two computations used different policies — value is always relative to a policyThe discount factor was applied in different directionsThe environment must be stochastic, so values fluctuate between runs

5. You have learned Q*(s,a) exactly for every state-action pair. How do you obtain an optimal policy?

Solve a system of linear equations built from Q*In every state, pick the action with the highest Q*-valueRun one more round of policy evaluation to extract the policyAverage Q* over actions to recover V*, then plan with the model

Next ChapterThe Bellman Equations→