Show Content

Function Approximation
Deep Q-Networks
DQN Improvements

Function Approximation DQN Improvements

Chapter 122

📝Draft

Deep Q-Networks

The breakthrough that made deep RL work

Prerequisites:

Q-Learning
Function Approximation

In this chapter

1 The DQN Architecture
2 Experience Replay
3 Target Networks
4 Putting It Together

What You'll Learn

Explain why naive Q-learning with neural networks fails
Describe how experience replay breaks correlation
Explain target networks and why they stabilize training
Implement DQN from scratch
Understand frame stacking and preprocessing for visual inputs

In 2013, a paper from DeepMind shook the AI world: a single algorithm, with the same hyperparameters, learned to play seven different Atari games from raw pixels, outperforming human experts on three of them. The expanded 2015 Nature version scaled the same recipe to 49 games, many at superhuman level.

That algorithm was DQN, and it showed that deep learning and reinforcement learning could work together.

The Breakthrough

📖Deep Q-Network (DQN)

A Q-learning agent that uses a deep neural network to approximate the Q-function, stabilized by two key innovations: experience replay and target networks.

The deadly triad (off-policy + function approximation + bootstrapping) seemed fatal. DQN survives by breaking two correlations:

Experience replay breaks the correlation between consecutive samples
Target networks break the correlation between Q-values and their targets

Chapter Overview

This chapter covers the complete DQN algorithm, piece by piece:

The DQN Architecture

CNNs for processing visual observations

Experience Replay

Breaking correlations through random sampling

Target Networks

Stabilizing training with frozen targets

Putting It Together

The complete DQN algorithm

The Core Idea

DQN is fundamentally just Q-learning with a neural network:

$Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]$

But instead of a table, we have:

A neural network $Q(s, a; \theta)$ parameterized by weights $\theta$
A replay buffer storing past experiences
A target network $Q(s, a; \theta^-)$ with frozen weights

The same machinery works far beyond Atari. On CartPole—our running example—a DQN with a two-layer fully connected network learns to balance the pole from four state variables; on Atari, only the network (a CNN) and the preprocessing change.

∑Mathematical Details

The DQN loss function:

$L(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[ \left( r + \gamma \max_{a'} Q(s', a'; \theta^-) - Q(s, a; \theta) \right)^2 \right]$

where $\mathcal{D}$ is the replay buffer and $\theta^-$ are the target network parameters.

Prerequisites

This chapter builds on:

Q-Learning for the core algorithm
Function Approximation for why we need neural networks and the challenges they introduce

Key Questions We’ll Answer

Why does naive neural network Q-learning fail?
How does storing and replaying experiences help?
Why do we need a separate target network?
What preprocessing is needed for visual inputs?

Check Your Understanding

Check your understanding

1. Why does naive Q-learning with a neural network tend to fail without DQN's additions?

Neural networks cannot represent Q-functions accurately enoughCorrelated sequential samples and targets that move with every update destabilize trainingBackpropagation cannot compute gradients through the max operatorEpsilon-greedy exploration does not work with neural networks

2. What does experience replay guarantee?

Convergence to the optimal Q-function, like tabular Q-learningNothing formally—it decorrelates samples and reuses data, which stabilizes training in practiceThat every transition is used exactly onceThat the training distribution exactly matches the current policy's distribution

3. Why does DQN use a separate target network instead of computing targets with the online network?

To reduce overestimation bias from the max operatorTo double the effective capacity of the Q-functionTo hold the regression target fixed for many steps, preventing the network from chasing its own updatesTo allow parallel training on two GPUs

4. Why does DQN stack 4 consecutive frames as its network input on Atari?

To increase the effective batch sizeA single frame hides motion—stacking restores velocity information the Markov state needsTo reduce memory usage in the replay bufferBecause CNNs require at least 4 input channels

5. Which statement about DQN's Atari results is accurate?

DQN beat human experts on all 49 games in the Nature paperDQN needed per-game hyperparameter tuning to reach human levelWith one algorithm and one set of hyperparameters, DQN reached superhuman performance on 29 of 49 gamesDQN solved sparse-reward games like Montezuma's Revenge

DQN = Q-learning + neural network + two stabilizers
The update rule is unchanged; experience replay and target networks make it trainable.
Experience replay decorrelates and reuses data
Random sampling from a buffer approximates i.i.d. training and lets each transition teach the network many times.
Target networks freeze the regression target
Between periodic syncs, training is supervised regression toward fixed targets instead of chasing a moving one.
Frame stacking restores the Markov property
Four stacked frames give the network motion information a single image lacks.
One recipe, 49 games
Identical architecture and hyperparameters reached superhuman play on 29 of 49 Atari games—the generality mattered as much as the scores.

Primary Sources

Mnih et al. (2013), “Playing Atari with Deep Reinforcement Learning,” NIPS Deep Learning Workshop — the original 7-game DQN paper. arXiv:1312.5602
Mnih et al. (2015), “Human-level control through deep reinforcement learning,” Nature 518 — the 49-game version with target networks, the standard DQN reference. Nature page

Next ChapterDQN Improvements→