REINFORCE and the Policy Gradient Theorem

We want to improve our policy by gradient ascent. But there’s a problem: the gradient of expected return involves the environment dynamics, which we don’t know. The Policy Gradient Theorem provides an elegant solution - we can compute the gradient using only samples from our policy.

Why REINFORCE?

In value-based methods like Q-learning, we learned a value function and derived a policy from it. REINFORCE takes a fundamentally different approach: optimize the policy directly.

The key insight is that we can compute policy gradients without knowing how the environment works. We just need to:

1. Sample trajectories from our current policy
2. Compute returns for those trajectories
3. Update the policy to make high-return actions more likely

Chapter Overview

This chapter derives the Policy Gradient Theorem and introduces REINFORCE, the simplest policy gradient algorithm. We’ll also tackle its main weakness - high variance - and introduce baselines as a solution.

The Big Picture

REINFORCE follows a simple recipe:

1. Collect a complete episode using the current policy
2. Compute the return (cumulative reward) from each timestep
3. Update the policy to increase the probability of actions that led to high returns

The Policy Gradient Theorem shows us that this gradient has a beautiful form:

$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot G_t \right]$

High-return actions get reinforced; low-return actions get suppressed. That’s the essence of REINFORCE.

Prerequisites

This chapter assumes familiarity with:

• The policy gradient objective from Introduction to Policy Gradients
• Stochastic policies and probability distributions over actions
• Basic calculus (gradients, chain rule)