rlbook.ai
Chapter 203
📝Draft

Actor-Critic Methods

Combining policy and value learning for stability

Prerequisites:

What You'll Learn

  • Explain the actor-critic architecture and why it helps
  • Define the advantage function and understand its benefits
  • Implement Advantage Actor-Critic (A2C) from scratch
  • Understand how TD learning provides bootstrap targets
  • Navigate the bias-variance tradeoff in actor-critic methods

REINFORCE taught us to weight log-probabilities by returns. But waiting for episode end is slow, and returns are noisy. What if we had a critic - a value function that could estimate how good a state is? We could get feedback every step, not just at episode end.

Why Actor-Critic?

Actor-critic methods combine the best of both worlds:

  • The Actor (policy) learns which actions to take
  • The Critic (value function) learns how good states are

The critic provides low-variance estimates of action quality, enabling the actor to learn faster and more stably than pure REINFORCE.

Chapter Overview

This chapter introduces actor-critic methods, the workhorse architecture of modern deep RL. We’ll cover the advantage function, A2C, and techniques for balancing bias and variance.

The Actor-Critic Idea

Two networks working together

Advantage Functions

How much better is this action than average?

Advantage Actor-Critic (A2C)

Synchronous actor-critic training

Generalized Advantage Estimation

Balancing bias and variance in advantage estimation

The Big Picture

Think of a student (actor) and a teacher (critic):

  • The student tries actions and learns from feedback
  • The teacher evaluates situations and provides guidance

The teacher doesn’t tell the student exactly what to do, but says “that situation looked promising” or “you were in trouble there.” This guidance helps the student learn faster than trial-and-error alone.

📖Actor-Critic

A family of algorithms where the actor learns a policy πθ(a∣s)\pi_\theta(a|s) and the critic learns a value function Vϕ(s)V_\phi(s). The critic’s estimates guide the actor’s learning.

The advantage function captures “how much better than average” an action is:

Aπ(s,a)=Qπ(s,a)−Vπ(s)A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)

We’ll keep training on CartPole, our running example: the A2C section solves it with per-step updates, so you can compare its learning curve directly against the episode-at-a-time REINFORCE agent from the previous chapter.

Prerequisites

This chapter assumes familiarity with:

  • The Policy Gradient Theorem from REINFORCE
  • Baselines and variance reduction
  • TD learning concepts (helpful but not required)

Check Your Understanding

Check your understanding
1. What do the actor and the critic each learn?
2. How does moving from REINFORCE to actor-critic change the bias-variance profile of the gradient estimate?
3. Why is the TD error delta = r + gamma*V(s') - V(s) a sensible estimate of the advantage A(s,a)?
4. What does GAE's lambda parameter control?
5. A2C can update every step instead of waiting for the episode to end. What makes that possible?
  1. Two learners, one loop
    The actor (policy) chooses actions; the critic (value function) evaluates outcomes and shapes the actor's gradient.
  2. Advantage = relative action quality
    A(s,a) = Q(s,a) - V(s): positive means better than this state's average, negative means worse.
  3. The TD error is a cheap advantage estimate
    delta = r + gamma*V(s') - V(s) approximates the advantage from a single transition—no full episode required.
  4. Bootstrapping trades bias for variance
    Unlike a pure baseline (unbiased), replacing returns with r + gamma*V(s') is biased while V is imperfect—and much less noisy.
  5. GAE puts the tradeoff on a dial
    One lambda parameter sweeps from one-step TD to Monte Carlo; 0.95 is the standard default.

Primary Sources

  • Mnih et al. (2016), “Asynchronous Methods for Deep Reinforcement Learning,” ICML 2016 — the A3C paper; A2C is its synchronous variant. arXiv:1602.01783
  • Schulman, Moritz, Levine, Jordan & Abbeel (2016), “High-Dimensional Continuous Control Using Generalized Advantage Estimation,” ICLR 2016 — introduces GAE and the lambda parameter. arXiv:1506.02438
  • Sutton, McAllester, Singh & Mansour (1999), “Policy Gradient Methods for Reinforcement Learning with Function Approximation,” NeurIPS 1999 — grounds the use of learned critics in the policy gradient theorem
Next ChapterProximal Policy Optimization→