Noisy Networks for Exploration

Prerequisites: Deep Q-Networks, DQN Improvements

The Problem: Exploration by Coin Flip

DQN explores with epsilon-greedy: with probability $\epsilon$ , ignore everything you’ve learned and act uniformly at random. It works, but it’s blunt:

State-blind. The exploration rate is the same in a state you’ve visited ten thousand times and one you’ve never seen.
Uninformative. A single random action rarely produces a meaningfully different trajectory—deep exploration requires sequences of coordinated deviations.
Another schedule to tune. You must hand-design how $\epsilon$ decays, and the right schedule differs per environment.

Fortunato et al. propose a different mechanism: put learnable noise on the network’s weights themselves. Because a perturbed weight affects the policy in every state, weight noise induces rich, temporally consistent variations in behavior—and because the noise scale is a trainable parameter, gradient descent decides how much randomness the agent needs, and where.

The Core Idea: Noisy Linear Layers

A NoisyNet replaces some of the network’s linear layers with noisy linear layers:

$y = (\mu^W + \sigma^W \odot \epsilon^W) x + (\mu^b + \sigma^b \odot \epsilon^b)$

$\mu^W, \mu^b$ are the usual mean weights and biases
$\sigma^W, \sigma^b$ are learnable noise scales
$\epsilon^W, \epsilon^b$ are zero-mean random samples, redrawn periodically

The network learns where to explore. If perturbing a weight consistently hurts returns, gradients shrink its $\sigma$ ; if the extra variation is harmless or useful for discovering rewards, $\sigma$ stays large. Exploration becomes state-dependent and self-annealing—no epsilon schedule, no entropy bonus.

One subtlety: the noise sample is held fixed across each forward pass (and in practice across steps between updates), so the perturbed network is a consistent alternative policy for a while, not per-action jitter. That consistency is what makes weight noise a form of deep exploration, closer in spirit to Thompson sampling than to epsilon-greedy.

∑Mathematical Details

Factorized Gaussian noise. The obvious approach samples independent noise for every weight:

$\epsilon_{ij} \sim \mathcal{N}(0, 1)$

which costs $p \times q$ random draws for a layer with $p$ inputs and $q$ outputs. The paper’s factorized variant uses one noise vector per side:

$\epsilon_{ij} = f(\epsilon_i) f(\epsilon_j), \qquad \epsilon_i, \epsilon_j \sim \mathcal{N}(0, 1), \qquad f(x) = \text{sign}(x)\sqrt{\lvert x \rvert}$

reducing the sample count to $p + q$ with negligible performance difference. Rainbow uses the factorized form; the paper’s A3C variant uses independent noise (noise sampling is cheap relative to A3C’s unrolled computation).

Initialization (factorized case):

$\mu$ entries uniform in $[-\frac{1}{\sqrt{p}}, \frac{1}{\sqrt{p}}]$
$\sigma$ entries set to $\frac{\sigma_0}{\sqrt{p}}$ with $\sigma_0 = 0.5$

Training. The loss is the expectation over noise of the usual RL loss, $\bar{L}(\zeta) = \mathbb{E}[L(\theta)]$ where $\theta = \mu + \sigma \odot \epsilon$ and $\zeta = (\mu, \sigma)$ . In practice a single noise sample gives an unbiased gradient estimate for both $\mu$ and $\sigma$ —standard backpropagation through the reparameterized weights.

For NoisyNet-DQN, the online network and the target network hold independent noise samples, and the loss’s inner and outer noise draws are independent as well.

Implementation

</>Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class NoisyLinear(nn.Module):
    """
    Noisy linear layer with factorized Gaussian noise.

    Replaces standard linear layer with learnable noise for exploration.
    """

    def __init__(self, in_features: int, out_features: int, sigma_init: float = 0.5):
        super().__init__()

        self.in_features = in_features
        self.out_features = out_features
        self.sigma_init = sigma_init

        # Learnable parameters: mean weights and biases
        self.weight_mu = nn.Parameter(torch.empty(out_features, in_features))
        self.bias_mu = nn.Parameter(torch.empty(out_features))

        # Learnable parameters: noise scales
        self.weight_sigma = nn.Parameter(torch.empty(out_features, in_features))
        self.bias_sigma = nn.Parameter(torch.empty(out_features))

        # Buffers for noise (not parameters, but saved in state_dict)
        self.register_buffer('weight_epsilon', torch.empty(out_features, in_features))
        self.register_buffer('bias_epsilon', torch.empty(out_features))

        self.reset_parameters()
        self.reset_noise()

    def reset_parameters(self):
        """Initialize parameters."""
        mu_range = 1 / math.sqrt(self.in_features)
        self.weight_mu.data.uniform_(-mu_range, mu_range)
        self.bias_mu.data.uniform_(-mu_range, mu_range)

        sigma_init = self.sigma_init / math.sqrt(self.in_features)
        self.weight_sigma.data.fill_(sigma_init)
        self.bias_sigma.data.fill_(sigma_init)

    def reset_noise(self):
        """Sample new noise."""
        epsilon_in = self._factorized_noise(self.in_features)
        epsilon_out = self._factorized_noise(self.out_features)

        # Outer product for factorized noise
        self.weight_epsilon.copy_(epsilon_out.outer(epsilon_in))
        self.bias_epsilon.copy_(epsilon_out)

    def _factorized_noise(self, size: int) -> torch.Tensor:
        """Generate factorized Gaussian noise: f(x) = sign(x) * sqrt(|x|)"""
        x = torch.randn(size, device=self.weight_mu.device)
        return x.sign() * x.abs().sqrt()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Forward pass with noisy weights."""
        if self.training:
            weight = self.weight_mu + self.weight_sigma * self.weight_epsilon
            bias = self.bias_mu + self.bias_sigma * self.bias_epsilon
        else:
            # No noise during evaluation
            weight = self.weight_mu
            bias = self.bias_mu

        return F.linear(x, weight, bias)


class NoisyDQN(nn.Module):
    """DQN with noisy layers for exploration."""

    def __init__(self, state_dim: int, n_actions: int, hidden_dim: int = 128):
        super().__init__()

        # Standard layers for feature extraction
        self.feature_layer = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU()
        )

        # Noisy layers for value prediction
        self.noisy1 = NoisyLinear(hidden_dim, hidden_dim)
        self.noisy2 = NoisyLinear(hidden_dim, n_actions)

    def forward(self, state: torch.Tensor) -> torch.Tensor:
        features = self.feature_layer(state)
        x = F.relu(self.noisy1(features))
        q_values = self.noisy2(x)
        return q_values

    def reset_noise(self):
        """Reset noise in all noisy layers."""
        self.noisy1.reset_noise()
        self.noisy2.reset_noise()


def demonstrate_noisy_exploration():
    """Show how noise affects action selection."""

    net = NoisyDQN(state_dim=4, n_actions=3)
    state = torch.randn(1, 4)

    # Multiple forward passes with same state but different noise
    print("Q-values with different noise samples:")
    for i in range(5):
        net.reset_noise()
        q_values = net(state)
        action = q_values.argmax().item()
        print(f"  Trial {i+1}: Q = {q_values.detach().numpy().round(3)}, Action = {action}")

    # Evaluation mode (no noise)
    net.eval()
    q_values_eval = net(state)
    print(f"\nEvaluation mode (no noise): Q = {q_values_eval.detach().numpy().round(3)}")

Practical notes:

Only replace the fully connected head. Convolutional feature layers stay deterministic; noise goes in the value/advantage streams. This keeps perception stable while decisions explore.
Act greedily with respect to the noisy network. There is no epsilon anymore—the noise is the exploration. Resample noise between environment steps (or per episode) so behavior varies.
Drop other exploration machinery. NoisyNet-DQN removes epsilon-greedy; NoisyNet-A3C removes the entropy bonus.

Results

Limitations and Extensions

Extra parameters and compute. Every noisy layer doubles its parameter count ( $\mu$ and $\sigma$ ), though the overhead is small next to conv layers.
Noise can decay prematurely. Since $\sigma$ is trained to reduce the loss, the agent can learn to switch exploration off before it has found the reward it needed exploration to find.
Related lineages: parameter-space noise for exploration (Plappert et al., 2018) reached similar conclusions from an evolutionary-strategies angle; randomized value functions and bootstrapped DQN (Osband et al., 2016) pursue the same “consistent random policy” idea with ensembles instead of weight noise.

Key Takeaways

Exploration becomes a learned quantity. Noise scales $\sigma$ are ordinary parameters trained by the same gradients as everything else.
Weight noise beats action noise for consistency. A fixed noise sample defines a coherent perturbed policy across many states, enabling deeper exploration than per-step random actions.
Factorized Gaussian noise makes it cheap: $p + q$ samples per layer instead of $p \times q$ .
It composes cleanly. Noisy layers slot into DQN, Dueling, A3C, and Rainbow without changing the loss or the algorithm structure.

Discussion Questions

Epsilon-greedy explores identically in every state; noisy nets explore state-dependently. Construct a small environment where this difference should matter, and predict each method’s behavior.
The $\sigma$ parameters are trained to minimize the RL loss. Why might that objective under-explore in sparse-reward environments? What auxiliary signal could fix it?
Why does holding a noise sample fixed for a while produce “deeper” exploration than resampling every step? What does this share with Thompson sampling?
In Rainbow’s ablation, removing noisy nets had a mixed, game-dependent effect. What properties of a game would make learned exploration noise help or hurt?