SARSA

On-policy TD control: learning while following your current policy

Prerequisites:

Introduction to TD Learning

What You'll Learn

Explain the transition from prediction to control
Implement SARSA for learning action values
Explain what “on-policy” means and its implications
Recognize when SARSA’s cautious behavior is desirable
Train an agent to solve GridWorld using SARSA

TD(0) learns $V^\pi$ —the value of states under a policy. But to control behavior, we need to evaluate actions, not just states. SARSA is TD for action values—and it gets its name from the quintuple $(S, A, R, S', A')$ that defines each update.

Why SARSA?

SARSA extends TD learning to control problems by learning action-values $Q(s,a)$ instead of state-values $V(s)$ . The key insight is that SARSA learns the value of the policy it’s actually following, including the exploration. This makes it “safe”—it learns to avoid dangerous states even if there’s a risky shortcut.

📖SARSA

An on-policy TD control algorithm that learns action values. The name comes from the tuple $(S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1})$ used in each update. SARSA learns the value of the policy it follows, including exploration.

The SARSA Update

∑Mathematical Details

The SARSA update rule:

$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[ R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t) \right]$

Notice the key difference from Q-learning (which we’ll see next): SARSA uses the actual next action $A_{t+1}$ , not the best possible action. This makes SARSA an on-policy method.

</>Implementation

def sarsa_update(Q, state, action, reward, next_state, next_action,
                 alpha=0.1, gamma=0.99, done=False):
    """Single SARSA update."""
    if done:
        td_target = reward
    else:
        td_target = reward + gamma * Q[next_state][next_action]

    td_error = td_target - Q[state][action]
    Q[state][action] += alpha * td_error
    return td_error

Chapter Overview

This chapter introduces SARSA—the first TD control algorithm. We’ll cover:

From Prediction to Control

Why we need action values and how GPI works with TD

The SARSA Algorithm

Complete algorithm with implementation and worked examples

On-Policy Behavior

The Cliff Walking example and why SARSA is safe

Prerequisites

This chapter assumes you’re comfortable with:

Introduction to TD Learning — TD(0) and the TD error
The RL Framework — Value functions and policies

Key Questions We’ll Answer

Why do we need action values for control?
What does “on-policy” mean and why does it matter?
Why is SARSA considered “safe” or “cautious”?
When should you choose SARSA over other methods?

The Big Picture

SARSA sits at a crucial point in the RL landscape:

It’s the first control algorithm we’ve seen that learns from experience
Unlike policy evaluation, it can actually find good policies
Unlike Q-learning (coming next), it accounts for exploration in its values

The name itself encodes the algorithm: at each step, you have State, take Action, get Reward, enter new State, and pick the next Action. Then you update.

From	To	What Changes
TD(0)	SARSA	State values → Action values
DP	SARSA	Model-based → Model-free
MC Control	SARSA	Episode end → Every step

Key Takeaways

SARSA learns action values with TD updates
Q(s,a) instead of V(s)—the extension that turns prediction into control.
The name encodes the update
Each update consumes the quintuple (S, A, R, S′, A′): state, action, reward, next state, and the actually-chosen next action.
On-policy: it evaluates the policy it follows
The values reflect epsilon-greedy behavior, exploration mistakes included.
SARSA is safe because exploration risk is priced in
States where a random action can lead to disaster get lower values, steering the policy away.
It learns Q^π for the exploratory policy, not Q*
Only if epsilon decays to zero (slowly enough—GLIE) does SARSA converge to the optimal values.
Choose SARSA when mistakes during training are costly
Robotics, finance, anything where the learning process itself must stay safe.

Check Your Understanding

Check your understanding

1. What does it mean that SARSA is "on-policy"?

It learns the value of the optimal policy while following an exploratory oneIt learns the value of the policy it is actually following, exploration includedIt can only be used with a deterministic policyIt updates the policy after every episode instead of every step

2. SARSA is trained on Cliff Walking with a fixed ε = 0.1. What path does its learned policy take, and why?

The shortest path along the cliff edge, because that maximizes rewardA random path, because ε-greedy never convergesThe longer, safer path away from the cliff, because its values include the risk that a random exploratory action pushes it off the edgeIt never reaches the goal with exploration enabled

3. What does SARSA converge to when trained with a fixed ε = 0.1?

Q*, the optimal action-value functionQ^π for the ε-greedy policy—values that account for 10% random actionsNothing—SARSA only converges when ε = 0The same values Q-learning converges to

4. Under what condition does SARSA converge to the optimal policy?

Always—any TD method finds the optimal policy eventuallyOnly if the environment is deterministicWhen the policy is GLIE: exploration decays to greedy slowly enough that every state-action pair is still visited infinitely often (plus standard step-size conditions)When ε is cut to exactly zero after a fixed number of episodes

Next ChapterQ-Learning→