What You'll Learn
- Explain why large policy updates are dangerous
- Understand the trust region concept and its importance
- Implement PPO with the clipped surrogate objective
- Tune PPO hyperparameters effectively
- Train agents using PPO on standard benchmarks
Policy gradient methods are powerful, but fragile. One bad update can destroy a good policy. TRPO solved this elegantly but with complex math. Then in 2017, OpenAI introduced PPO - a simpler heuristic that performs comparably in practice. Today, PPO is one of the most widely used deep RL algorithms, especially in on-policy settings.
Why PPO?
The core insight is simple: discourage the new policy from moving too far from the old one.
- TRPO enforces this with a KL divergence constraint solved via second-order optimization
- PPO instead uses a clipped objective that removes the incentive for large per-sample updates and can be optimized with standard gradient descent (it discourages, but does not bound, policy change — practitioners monitor KL as the safeguard)
The result: simpler code, similar performance, and remarkable robustness.
Chapter Overview
This chapter explains PPO, the workhorse algorithm behind modern RL applications from game-playing agents to RLHF for language models.
Trust Regions
Why we need to limit how much policies can change
The PPO Algorithm
Clipped surrogate objectives explained
Why PPO Works
Simplicity, stability, and performance
PPO in Practice
Hyperparameters, tricks, and common pitfalls
The Big Picture
A policy gradient algorithm that discourages large policy updates by clipping the surrogate objective when the probability ratio between new and old policies leaves a small interval. Samples pushed outside the interval stop contributing gradient, removing the incentive for destructively large updates while maintaining simplicity.
The PPO clipped objective:
where is the probability ratio.
Think of PPO as a “cautious optimizer”:
- If an action looks good (positive advantage), increase its probability - but stop rewarding increases beyond the clip range
- If an action looks bad (negative advantage), decrease its probability - but stop rewarding decreases beyond the clip range
- The clip removes the incentive to overreact to any single batch of experience
Prerequisites
This chapter assumes familiarity with:
- The advantage function from Actor-Critic Methods
- GAE for advantage estimation
- Basic policy gradient concepts
Key Takeaways
- Large policy updates can be catastrophicOne bad update can destroy a good policy. TRPO constrains updates with a trust region; PPO discourages them with clipping.
- PPO replaces second-order optimization with clippingTRPO needs conjugate gradients and line search; PPO runs on plain gradient descent.
- Clipping removes incentive; it does not bound changeA clipped sample stops contributing gradient, but other samples still move the shared parameters—ratios and KL can exceed the clip range. Monitor KL in practice.
- PPO is robust across domainsForgiving hyperparameters and stable training made it the default for games, robotics, and more.
- PPO powers RLHF for language modelsThe InstructGPT pipeline—precursor to ChatGPT—used PPO as its RL algorithm.