State Value Functions | Value Functions

The most fundamental question in reinforcement learning: How good is it to be in this state?

The state-value function answers this question precisely. Imagine you’re playing a board game and look at the current position: “Am I winning or losing?” That intuitive assessment is what $V^\pi(s)$ formalizes—it compresses all future possibilities into one number.

📖State-Value Function

The state-value function $V^\pi(s)$ is the expected return—the discounted sum of all future rewards—when starting in state $s$ and following policy $\pi$ thereafter.

∑Mathematical Details

$V^\pi(s) = \mathbb{E}_\pi[G_t | S_t = s]$

where $G_t$ is the discounted return from time $t$ .

Think of value as a “how close to treasure” measure. In a navigation problem:

States near the goal have high values because rewards are within reach
States far from the goal have lower values because rewards are distant (and discounted)
States in dead ends or dangerous areas have very low values

The values form a gradient that essentially “points toward” the rewards. If you could see the value of every state, you’d see a landscape with peaks at rewarding states.

Values Depend on Policy

A crucial insight: value is always relative to a policy. The same state can have very different values under different policies.

📌Policy Changes Everything

Consider a simple 3-state problem:

[Start] --right--> [Middle] --right--> [Goal: +10]
   |                   |
   v                   v
  wall               [Pit: -10]

Rewards arrive on transitions: +10 for entering Goal, $-10$ for entering Pit, 0 otherwise. Goal and Pit are terminal, so their values are 0. Discount: $\gamma = 0.9$ .

Under a smart policy (always go right from Start, always go right from Middle):

$V^\pi(\text{Start}) = 0 + 0.9 \times 10 = 9.0$ (the +10 arrives one step later, discounted once)
$V^\pi(\text{Middle}) = 10.0$ (the +10 arrives on the very next transition)
$V^\pi(\text{Goal}) = 0$ (terminal: no future rewards)

Under a terrible policy (always go down from Middle):

$V^\pi(\text{Start}) = 0 + 0.9 \times (-10) = -9.0$ (leads to the pit one step later)
$V^\pi(\text{Middle}) = -10.0$ (goes straight to the pit)
$V^\pi(\text{Goal}) = 0$ (terminal, unchanged)

Same states, drastically different values. The policy determines the value.

The Full Definition

The value is an expected return because two sources of randomness stand between you and the future: the policy may choose actions stochastically, and the environment may respond stochastically. Even from the same state, following the same policy, you can experience different trajectories—the value averages over all of them.

∑Mathematical Details

Expanding the definition with the return formula:

$V^\pi(s) = \mathbb{E}_\pi\left[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} \bigg| S_t = s\right]$

The expectation is taken over:

The stochastic policy $\pi(a|s)$ : Which actions we might take
The stochastic environment $P(s'|s,a)$ : Where those actions might lead
All future time steps: The infinite sum of discounted rewards

This makes $V^\pi$ a well-defined function for any policy $\pi$ in any MDP with bounded rewards and $\gamma < 1$ .

GridWorld Example

Let’s compute values for a concrete example. Consider this 4x4 GridWorld:

📌GridWorld Value Computation

 _____ _____ _____ _____
|     |     |     |     |
| S   |  .  |  .  |  .  |
|_____|_____|_____|_____|
|     |     |     |     |
|  .  |  X  |  .  |  .  |
|_____|_____|_____|_____|
|     |     |     |     |
|  .  |  .  |  .  |  .  |
|_____|_____|_____|_____|
|     |     |     |     |
|  .  |  .  |  .  |  G  |
|_____|_____|_____|_____|

S = Start, G = Goal (+10), X = Wall, . = Empty (-1 per step)

Reward setup (used for both tables below): every move costs $-1$ , including moves that bump into a wall or the grid edge (the agent stays in place but still pays). Entering G additionally pays +10 on that transition, so the final step nets $+9$ . G is terminal, so $V(\text{G}) = 0$ . Discount: $\gamma = 0.9$ .

Under a random policy (equal probability for each direction):

	Col 0	Col 1	Col 2	Col 3
Row 0	-8.9	-8.6	-7.8	-7.2
Row 1	-8.6	Wall	-6.4	-5.5
Row 2	-7.8	-6.4	-4.3	-0.9
Row 3	-7.2	-5.5	-0.9	0.0

Values are negative everywhere: the random policy wanders so long that the accumulated step penalties outweigh the (heavily discounted) +10 bonus, even one step from the goal.

Under an optimal policy (always move toward goal):

	Col 0	Col 1	Col 2	Col 3
Row 0	1.22	2.47	3.85	5.39
Row 1	2.47	Wall	5.39	7.10
Row 2	3.85	5.39	7.10	9.00
Row 3	5.39	7.10	9.00	0.00

Now values increase as we approach the goal: 9.00 one step away (the netted $+9$ arrives immediately), then $-1 + 0.9 \times 9.00 = 7.10$ two steps away, and so on. The value gradient points directly toward the reward. The goal cell itself shows 0.00 because it is terminal; the +10 lives on the transitions into it.

How Discount Factor Affects Values

The discount factor $\gamma$ dramatically changes the value landscape:

💡Tip

Think of $\gamma$ as “how far-sighted” the agent is:

$\gamma = 0$ : Completely myopic. Only immediate rewards matter.
$\gamma = 0.9$ : Moderate foresight. Rewards 10 steps away worth about 35% of immediate.
$\gamma = 0.99$ : Far-sighted. Rewards 100 steps away still worth about 37%.
$\gamma = 1$ : No discounting; all future rewards count fully. Only safe for episodic tasks that are guaranteed to terminate.

📌Discount Factor Impact

Consider a state that’s 5 steps from a +100 reward (with no intermediate rewards). The value of that state is the +100 discounted five times—and the choice of $\gamma$ changes it enormously:

3.125

γ = 0.5

59.0

γ = 0.9

95.1

γ = 0.99

Each number is $100 \times \gamma^5$ . Higher $\gamma$ means distant states still have substantial value. Lower $\gamma$ means only nearby rewards matter.

Computing Values by Hand

For simple MDPs, we can compute values directly from the definition.

📌Manual Value Calculation

Consider a 3-state chain with deterministic policy $\pi$ that always goes right:

[A] --right--> [B] --right--> [C: terminal]

Rewards on transitions: A->B gives -1, B->C gives +10
Discount: γ = 0.9

Work backward from the terminal state. C is terminal, so $V^\pi(C) = 0$ : there are no rewards after reaching C, and the +10 is earned on the transition into it. B is one step from C, so it collects the +10 immediately: $V^\pi(B) = 10.0$ . A pays the $-1$ step cost, then gets B’s value discounted once: $V^\pi(A) = 8.0$ .

∑Mathematical Details

State C (terminal): $V^\pi(C) = 0$

State B: $V^\pi(B) = R(B \to C) + \gamma V^\pi(C) = 10 + 0.9 \times 0 = 10.0$

State A: $V^\pi(A) = R(A \to B) + \gamma V^\pi(B) = -1 + 0.9 \times 10.0 = 8.0$

The values form a gradient: $8.0 \to 10.0$ , increasing as we approach the goal.

Values as Predictions

ℹ️Note

Value functions are predictions. They predict the expected cumulative reward. Good predictions enable good decisions: if you know the value of every state you could end up in, you can evaluate any policy.

This predictive nature is why value functions are central to RL:

Evaluation: Given a policy, compute its value function to see how good it is
Improvement: Use values to find better policies
Learning: Estimate values from experience when the MDP is unknown

∑Mathematical Details

The predictive interpretation becomes clear when we think about Monte Carlo estimation. If we run many episodes following policy $\pi$ and track the returns from state $s$ :

$V^\pi(s) \approx \frac{1}{N} \sum_{i=1}^{N} G_t^{(i)}$

where $G_t^{(i)}$ is the return from the $i$ -th visit to state $s$ . As $N \to \infty$ , this converges to the true value.

Implementation

</>Implementation

Here’s how to represent and estimate value functions in Python:

import numpy as np
from collections import defaultdict

# Value function as a dictionary
V = defaultdict(float)

# Simple Monte Carlo value estimation
def estimate_values_mc(episodes, gamma=0.99):
    """
    Estimate V(s) from a list of episodes using Monte Carlo.

    Each episode is a list of (state, reward) tuples, where reward is
    the one received on the transition out of that state.
    """
    returns = defaultdict(list)

    for episode in episodes:
        # Calculate returns for each state visited
        G = 0
        # Work backward through the episode
        for state, reward in reversed(episode):
            G = reward + gamma * G
            returns[state].append(G)

    # Average the returns for each state
    V = {}
    for state, state_returns in returns.items():
        V[state] = np.mean(state_returns)

    return V


# Example usage (same chain MDP as above: A->B costs -1, B->C pays +10)
episodes = [
    # Episode 1: A -> B -> C (goal)
    [('A', -1), ('B', 10)],
    # Episode 2: A -> B -> C (goal)
    [('A', -1), ('B', 10)],
    # Episode 3: A -> A -> B -> C (got stuck briefly)
    [('A', -1), ('A', -1), ('B', 10)],
]

V = estimate_values_mc(episodes, gamma=0.9)
print("Estimated values:")
for state in sorted(V.keys()):
    print(f"  V({state}) = {V[state]:.2f}")

Output:

Estimated values:
  V(A) = 7.55
  V(B) = 10.00

Note that the estimate for state A (7.55) is below the true value of 8.0 we computed by hand, because episode 3 started poorly (stuck at A). The terminal state C never accrues a return; its value is 0 by convention.

Visualizing Value Functions

Values are often visualized as heatmaps, showing the “terrain” of the value landscape:

</>Implementation

import numpy as np
import matplotlib.pyplot as plt

def visualize_values(V, grid_shape, title="Value Function"):
    """
    Visualize a value function as a heatmap.

    V: dict mapping (row, col) -> value
    grid_shape: (rows, cols) tuple
    """
    rows, cols = grid_shape
    value_grid = np.zeros((rows, cols))

    for (r, c), value in V.items():
        value_grid[r, c] = value

    plt.figure(figsize=(8, 6))
    plt.imshow(value_grid, cmap='RdYlGn', interpolation='nearest')
    plt.colorbar(label='Value')

    # Add value labels to each cell
    for r in range(rows):
        for c in range(cols):
            plt.text(c, r, f'{value_grid[r, c]:.1f}',
                    ha='center', va='center', fontsize=12)

    plt.title(title)
    plt.xlabel('Column')
    plt.ylabel('Row')
    plt.show()


# Example: 4x4 GridWorld values under optimal policy
# (-1 per step, +10 on entering the goal, gamma = 0.9; see table above)
V_optimal = {
    (0, 0): 1.22, (0, 1): 2.47, (0, 2): 3.85, (0, 3): 5.39,
    (1, 0): 2.47, (1, 1): 0.0, (1, 2): 5.39, (1, 3): 7.10,  # (1,1) is wall
    (2, 0): 3.85, (2, 1): 5.39, (2, 2): 7.10, (2, 3): 9.00,
    (3, 0): 5.39, (3, 1): 7.10, (3, 2): 9.00, (3, 3): 0.0,  # goal (terminal)
}

visualize_values(V_optimal, (4, 4), "GridWorld Values (Optimal Policy)")

The resulting heatmap shows values increasing toward the goal in the bottom-right corner. The wall and the terminal goal cell both display zero.

Key Properties of Value Functions

Three facts make value functions mathematically well-behaved: they’re always finite (bounded rewards plus discounting keep the infinite sum in check), each policy has exactly one value function, and value functions let us rank policies against each other.

∑Mathematical Details

Property 1: Boundedness

For bounded rewards $|R| \leq R_{max}$ and $\gamma < 1$ :

$|V^\pi(s)| \leq \frac{R_{max}}{1 - \gamma}$

This ensures values are always finite and well-defined.

Property 2: Uniqueness

For a given policy $\pi$ and MDP, there is exactly one value function $V^\pi$ . Different policies have different value functions, but each policy determines a unique one.

Property 3: Policy Ordering

We can compare policies via their value functions. Policy $\pi$ is better than $\pi'$ if:

$V^\pi(s) \geq V^{\pi'}(s) \text{ for all } s \in \mathcal{S}$

Summary

State-value function $V^\pi(s)$ measures expected return from state $s$ under policy $\pi$
Values depend on the policy: same state, different policies, different values
Values form a gradient pointing toward rewards
The discount factor $\gamma$ controls how much future rewards matter
Values can be estimated from experience using Monte Carlo methods

But state values alone don’t tell us what to do. To make decisions, we need to compare actions. That’s where action-value functions come in.