Chapter 102: Contextual Bandits

What You'll Learn

Understand how contextual bandits extend multi-armed bandits with features
Formalize the contextual bandit problem mathematically
Implement LinUCB for linear reward models
Recognize real-world applications in recommendations, advertising, and personalization
Understand the bridge from bandits to full reinforcement learning

A news website needs to decide which headline to show each visitor. But here’s the catch: different people like different things. A sports fan wants game scores; a tech enthusiast wants startup news.

This isn’t just about finding the best arm—it’s about finding the best arm for each person. Welcome to contextual bandits.

From One-Size-Fits-All to Personalization

In multi-armed bandits, we sought a single best action. But many real problems have context—features that should inform our decision:

User profile: Age, location, browsing history
Time of day: Morning news vs evening entertainment
Device type: Mobile users want shorter content

Contextual bandits learn to map these features to actions, personalizing decisions while still exploring efficiently.

ℹ️Note

Contextual bandits are the workhorse behind modern recommendation systems, online advertising, and personalized medicine. They’re simpler than full RL but more powerful than basic bandits.

Chapter Overview

Linear UCB

Contextual exploration with linear models

Real-World Applications

Recommendations, ads, and clinical trials

The Key Insight

📖Contextual Bandit

A sequential decision problem where the agent observes context (features) before choosing an action, and the expected reward depends on both the context and the chosen action.

Contextual bandits bridge the gap between:

Simple bandits: One best arm for everyone
Full RL: Sequential decisions that change the environment

By conditioning on context, we personalize without needing the full complexity of states and transitions.

Prerequisites

This chapter builds on:

Exploration strategies from Multi-Armed Bandits
Especially UCB which LinUCB extends

Key Takeaways

Context allows personalization
Instead of one best arm for everyone, the agent learns a mapping from features to actions—different users get different recommendations.
LinUCB extends UCB to linear reward models
Confidence ellipsoids over the weight vector play the role that per-arm confidence intervals played in plain UCB.
Context is not state
Your action does not change the next context you see. That independence is exactly what keeps contextual bandits simpler than full RL.
Contextual bandits power real systems
News recommendation, online advertising, and adaptive clinical trials all run on these ideas.
When actions affect future context, you need full RL
Once today’s recommendation changes tomorrow’s user, you are in MDP territory—the subject of the next chapters.

Check Your Understanding

Check your understanding

1. What distinguishes context (in a contextual bandit) from state (in full RL)?

Context is continuous while state is discreteNothing—context and state are two names for the same thingContext arrives independently of the agent’s actions, while state transitions depend on the actions takenContext contains more features than a state does

2. How does LinUCB decide which arm to pull for a given context?

It predicts each arm’s reward from a linear model and adds an uncertainty bonus, then picks the highest sumIt picks the arm whose linear model has the smallest prediction error so farIt maintains a separate epsilon-greedy bandit for every distinct context vectorIt samples a random context and matches it to the nearest arm

3. A news site’s recommender changes what a user reads today, which changes what that same user wants tomorrow. What does this imply?

Nothing—contextual bandits handle this automaticallyThe problem has become sequential: actions affect future context, so a full RL (MDP) formulation is needed to account for itThe site should just use a larger context vectorThe site should switch back to a non-contextual bandit

4. Why not just learn each user segment’s best arm with a separate multi-armed bandit?

That approach is what LinUCB actually does internallySeparate bandits cannot share what they learn across similar contexts, so data is fragmented and rare contexts never get enough samplesSeparate bandits would have lower regret but are too slow to runMulti-armed bandits cannot handle more than a few arms

Next ChapterMonte Carlo Methods→

What You'll Learn

From One-Size-Fits-All to Personalization

Chapter Overview

Why Context Matters

Linear UCB

Real-World Applications

The Key Insight

Prerequisites

Key Takeaways

Check Your Understanding