Elevator Dispatch with Multi-Agent RL

You’re on the 8th floor, running late. You press the elevator button and wait. And wait.

An empty elevator passes by—heading somewhere else. When one finally arrives, it’s already packed.

Why does this happen?

Elevators solve a coordination problem: multiple agents working together to minimize wait times across an entire building, adapting to changing traffic patterns throughout the day.

This isn’t simple scheduling—it’s sequential decision-making under uncertainty. Passenger arrivals are random, demands shift (morning rush vs quiet periods), and elevators decide in real-time without knowing future requests.

Perfect for reinforcement learning.

See It In Action

Before diving into the details, try controlling elevators yourself! Compare different algorithms:

Elevator Dispatch Simulation

Building View

0/8

Metrics

Timestep

Waiting

Avg Wait (steps)

0.0

Max Wait (steps)

Delivered

0/0

Utilization

Algorithm

Traffic Pattern

Speed: 500ms

Legend: Blue = Low utilization, Yellow = Medium, Red = High | Orange dots = Waiting passengers | Wait times are simulation steps, not seconds

Note: All four modes are hand-coded dispatch heuristics — "Smart Heuristic" targets the floor with the most waiting passengers; it is not a trained RL policy. Running a trained in-browser RL policy is future work.

Quick experiment:

Set algorithm to “Nearest Car”, traffic to “Morning Rush”
Press Play and watch wait times
Switch to “Random” — notice the chaos!
Try “SCAN” and “Smart Heuristic” — compare how dispatch rules differ

ℹ️Note

This demo compares hand-coded dispatch heuristics so you can build intuition for the problem. It does not run a trained RL policy in the browser (that’s future work) — training an agent yourself is covered later on this page.

Now let’s understand why this problem is challenging…

Why Traditional Rules Fail

Traditional elevator systems use simple heuristics:

First-Come-First-Served

Serve requests in arrival order

❌ Can starve upper floors

SCAN Algorithm

Continue in direction until no more requests

❌ Ignores individual wait times

Nearest Car

Send closest available elevator

❌ No lookahead

These work in simple scenarios but struggle when faced with:

Time-varying traffic — Rush hours vs quiet periods
Multi-objective tradeoffs — Wait time vs energy vs fairness
Coordination — Multiple elevators working together
Long-term planning — Positioning for future demand

RL agents learn policies that handle all of this.

The Problem Domain

Our Test Building

Floors

Elevators

Capacity

10m

Episode

Each episode simulates 300 timesteps (10 minutes at 2 seconds per step). Elevators move at 0.5 floors/timestep.

Traffic Patterns

Real buildings have predictable patterns throughout the day:

🌅

Morning Rush (7-9am)

70% lobby → upper floors, high volume

🍽️

Lunch Time (12-1pm)

Bidirectional, moderate volume

🌆

Evening Rush (5-7pm)

70% upper floors → lobby, high volume

🌙

Quiet Period

Low volume, random destinations

ℹ️Note

Passengers arrive following a Poisson process with time-varying rates (λ changes by pattern).

Success Metrics

We measure performance across multiple dimensions:

⏱️

Average wait time

Primary user experience metric

⚠️

Max wait time

Fairness — prevent starvation

📊

Throughput

How many passengers served

⚡

Energy cost

Total floors traveled (efficiency)

📈

Utilization

How full elevators are when moving

The RL agent must balance all of these simultaneously.

The MDP Formulation

State Space: What Each Elevator Sees

Each elevator has a 57-dimensional observation vector containing:

Own State (15 dims)

• Current floor (1)
• Direction UP/DOWN/IDLE (3, one-hot)
• Passenger count (1)
• Destination floors — which buttons pressed (10, binary)

Pending Requests (30 dims)

• Waiting passengers per floor (10)
• Up-button pressed per floor (10, binary)
• Down-button pressed per floor (10, binary)

Other Elevators (8 dims)

• Positions of other 2 elevators
• Their directions (3-way one-hot each)

Time Context (4 dims)

• Traffic pattern (one-hot)
• Morning/Lunch/Evening/Quiet

⚠️Warning

Partial observability: Elevators don’t know exact passenger destinations until they board—only which floors have waiting passengers and their desired direction.

∑Mathematical Details

The observation for elevator $i$ at time $t$ is:

$\mathbf{o}_i^t = [\mathbf{s}_i^t, \mathbf{r}^t, \mathbf{e}_{-i}^t, \mathbf{c}^t]$

where:

$\mathbf{s}_i^t \in \mathbb{R}^{15}$ : Own state (floor, direction one-hot, passenger count, destination floors binary)
$\mathbf{r}^t \in \mathbb{R}^{30}$ : Requests (waiting passengers per floor, up buttons, down buttons)
$\mathbf{e}_{-i}^t \in \mathbb{R}^{8}$ : Other elevators (2 elevators × 4 features each)
$\mathbf{c}^t \in \mathbb{R}^{4}$ : Traffic context (one-hot)

Total observation dimension: $d = 15 + 30 + 8 + 4 = 57$ (for 10 floors and 3 elevators — always read it from env.observation_space rather than hardcoding it)

Action Space: Where To Go Next

Each elevator chooses a target floor (0-9) every timestep.

💡Tip

We use high-level actions (target floor) rather than low-level control (MOVE_UP/MOVE_DOWN/OPEN_DOOR). The environment handles pathfinding—if target is floor 7, the elevator moves toward 7, stopping to pick up/drop off passengers en route.

Per Elevator

Discrete(10)

Joint (3 elevators)

10³ = 1000

∑Mathematical Details

At each timestep $t$ , each elevator $i$ selects an action:

$a_i^t \in \mathcal{A}_i = \{0, 1, \ldots, 9\}$

The joint action is $\mathbf{a}^t = (a_1^t, a_2^t, a_3^t)$ .

Reward Design: Balancing Multiple Goals

The reward function must balance competing objectives:

Wait Time Penalty

-1.0 per waiting passenger per timestep

Primary goal: minimize wait

Starvation Penalty

-5.0 per passenger waiting > 60 timesteps

Fairness: don’t ignore anyone

Delivery Bonus

+10.0 per passenger delivered

Reward completion, not just attempts

Energy Cost

-0.1 per floor moved

Efficiency: discourage wasted movement

💡Tip

Why is delivery bonus (+10) so much larger than wait penalty (-1)? Because it takes multiple timesteps to pick up and deliver a passenger. This ratio prevents short-term thinking.

Total reward each timestep:

$r^t = r_{\text{wait}} + r_{\text{starvation}} + r_{\text{delivery}} + r_{\text{energy}}$

∑Mathematical Details

Formally, the reward at timestep $t$ is:

$r^t = -\sum_{p \in W^t} w(p) + 10 \cdot |D^t| - 0.1 \cdot \sum_{i=1}^{3} m_i^t$

where:

$W^t$ : set of waiting passengers at time $t$
$w(p)$ $w (p)$ : per-passenger wait penalty weight, defined piecewise:
- $w(p) = 6$ if the passenger’s wait time exceeds 60 timesteps (the base $-1$ plus the $-5$ starvation penalty)
- $w(p) = 1$ otherwise
$D^t$ : set of passengers delivered at time $t$
$m_i^t$ : floors moved by elevator $i$ at time $t$ (this timestep only, not cumulative)

The discount factor is $\gamma = 0.99$ (episodes are short, so we weight near-future heavily).

Episode Structure

Duration

300 steps

(10 min)

Start Position

Floor 0

(lobby)

Arrivals

Poisson

(λ varies)

Termination

Fixed

(no early end)

This creates episodes with rush hours, coordination challenges, and consequences for positioning decisions.

The Multi-Agent Challenge

With a single elevator, this is a standard MDP. With three elevators? Much harder.

1. Credit Assignment

When wait times improve, which elevator deserves credit? All three contribute, but actions are coupled.

2. Non-Stationarity

From elevator 1’s view, the world is constantly changing because elevators 2 and 3 are learning too. What worked yesterday fails today.

3. Coordination Without Communication

Elevators must learn to partition floors or specialize—without talking to each other. Coordination emerges from shared experience.

4. Safe Exploration

Trying “what if I skip this request?” can cause terrible wait times. Can’t explore recklessly in production.

∑Mathematical Details

This is a Decentralized Partially Observable Markov Decision Process (Dec-POMDP).

Formally:

Agents: $\mathcal{N} = \{1, 2, 3\}$ (three elevators)
Joint observation: $\mathbf{o}^t = (o_1^t, o_2^t, o_3^t)$ where each $o_i^t$ depends on true state $s^t$
Joint action: $\mathbf{a}^t = (a_1^t, a_2^t, a_3^t)$
Shared reward: $r^t$ (all agents receive the same reward signal)
Transition: $s^{t+1} \sim P(\cdot | s^t, \mathbf{a}^t)$

The goal is to find a joint policy $\pi = (\pi_1, \pi_2, \pi_3)$ that maximizes expected return:

$J(\pi) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^{T} \gamma^t r^t\right]$

Baseline Approaches

Before applying RL, let’s see what simple rules achieve:

🎲 Random

Each elevator picks random floors.

✓ Dead simple

✗ Ignores all info

✗ Terrible

🚗 Nearest Car

Closest idle elevator takes each request.

✓ Intuitive

✗ Elevators cluster

✗ No lookahead

↕️ SCAN

Continue in direction until no more requests, then reverse.

✓ Good for uni-directional

✗ Poor for bidirectional

✗ No coordination

ℹ️Note

These baselines set performance bounds our RL agent must beat.

The RL Solution: Independent Q-Learning

Our approach: Independent Q-Learning with a shared replay buffer.

💡 Key Idea

Each elevator has its own Q-network learning

Q_i(o_i, a_i)

, but all three share a replay buffer.

✓ Decentralized execution — each elevator decides independently

✓ Implicit coordination — learn from each other’s experiences

✓ Simple — easier than QMIX, MADDPG, etc.

Network Architecture

Q-Network (one per elevator)

Input:57 dims (from obs space)

Hidden:[128, 128] ReLU

Output:10 Q-values

Training Hyperparameters

Replay buffer:50,000

Batch size:64

ε-greedy:1.0 → 0.01

Target update:every 100 steps

Optimizer:Adam lr=0.001

</>Implementation

Here’s the core training loop:

from rlbook.envs import ElevatorDispatch
from rlbook.agents import ElevatorDQN

# Create environment
env = ElevatorDispatch(
    n_floors=10,
    n_elevators=3,
    traffic_pattern="morning_rush",
    max_timesteps=300
)

# Derive the per-elevator observation size from the environment
# (never hardcode it -- it changes with n_floors and n_elevators)
obs_dim = env.observation_space["elevator_0"].shape[0]

# Create multi-agent DQN
agent = ElevatorDQN(
    n_floors=10,
    n_elevators=3,
    observation_dim=obs_dim,
    hidden_dims=(128, 128),
    gamma=0.99,
    epsilon=1.0,
    epsilon_decay=0.995
)

# Training loop
for episode in range(1000):
    obs, info = env.reset()
    episode_reward = 0

    for step in range(env.max_timesteps):
        # Each elevator selects action from its Q-network
        actions = agent.select_actions(obs, training=True)

        # Environment step
        next_obs, reward, done, truncated, info = env.step(actions)

        # Store all three elevators' transitions
        agent.store_transitions(obs, actions, reward, next_obs, done)

        # Train all networks
        loss = agent.train_step()

        episode_reward += reward
        obs = next_obs

        if done or truncated:
            break

    # Decay exploration
    agent.decay_epsilon()

Full implementation: code/rlbook/examples/train_elevator.py

∑Mathematical Details

Each elevator $i$ learns a Q-function $Q_i: \mathcal{O}_i \times \mathcal{A}_i \to \mathbb{R}$ via:

$Q_i(o_i^t, a_i^t) \leftarrow Q_i(o_i^t, a_i^t) + \alpha \left[r^t + \gamma \max_{a_i'} Q_i(o_i^{t+1}, a_i') - Q_i(o_i^t, a_i^t)\right]$

Note that the reward $r^t$ is shared (global), but each elevator updates based on its own observation-action pairs.

The policy for elevator $i$ is:

$\pi_i(o_i) = \arg\max_{a_i} Q_i(o_i, a_i)$

During training, we use ε-greedy exploration:

$a_i^t = \text{argmax}_{a_i} Q_i(o_i^t, a_i) \text{ with prob. } 1-\epsilon, \text{ else random}$

Or more precisely:

With probability $\epsilon$ : select random floor
With probability $1-\epsilon$ : select $\pi_i(o_i^t)$

Evaluating the Agent

How to compare policies fairly

When you train an agent (see Try It Yourself), evaluate it the way you would any RL result:

1. Same traffic, same seeds

Run every policy (Random, Nearest Car, SCAN, DQN) on the same set of evaluation episodes with fixed seeds, so all policies face identical passenger arrivals.

2. Compare the full metric set

Average wait (spawn → pickup, over served passengers), max wait (fairness), passengers served (throughput), and floors traveled (energy). A policy can win on one metric by sacrificing another.

3. Many episodes, exploration off

Evaluate with ε = 0 over enough seeded episodes to average out arrival randomness (the table below uses 20), and report variance, not just the mean.

4. Beat the strong baseline, not the weak one

SCAN is a genuinely good heuristic for uni-directional traffic. “Better than Random” is not evidence that RL is working; “consistently better than SCAN across traffic patterns” is.

Measured baseline results

The table below is measured, not illustrative. It was generated by code/rlbook/examples/evaluate_elevator.py running each baseline policy on the real ElevatorDispatch environment at its default configuration (10 floors, 3 elevators, morning-rush traffic, 300-step episodes): one episode per seed, seeds 1000–1019, so every policy faces identical passenger arrival streams. The rendered table comes straight from the committed artifact code/benchmarks/elevator_baselines.json via the script’s --markdown flag — benchmark numbers on this page are never hand-typed.

Policy	Avg wait (steps)	Max wait (steps)	Delivered	Floors traveled
Random	24.6 ± 10.3	296.0 ± 3.2	12.3 ± 4.7	426.3 ± 3.4
Nearest Car	9.1 ± 1.2	27.7 ± 3.4	123.2 ± 10.2	365.8 ± 9.2
SCAN	10.3 ± 2.6	32.2 ± 5.5	122.7 ± 10.0	360.1 ± 6.4
DQN (independent Q-learning)	pending training run

Mean ± std over 20 episodes (seeds 1000–1019, 300 steps each; 131.9 passengers spawned per episode on average).

What the numbers actually say, with no narrative forced on them:

Both heuristics crush Random: they serve roughly 123 of ~132 spawned passengers, while Random serves about 12 and starves the rest (its max wait is essentially the episode length).
Nearest Car edges out SCAN here on average wait (9.1 vs 10.3 steps) and max wait, while SCAN travels slightly fewer floors. Under morning-rush traffic most requests originate at the lobby, which suits Nearest Car’s oldest-request assignment; SCAN’s advantage of batching same-direction stops matters less when the implementation makes each elevator sweep toward the same shared request set (they partially herd). Different traffic patterns or a zoned SCAN could reorder these two — measure, don’t assume.
DQN has no numbers yet because we have not published a reproducible training run. A checkpoint exists at code/trained_models/elevator_dqn.pt, but it was trained before the 2026-07 environment fixes (observation space, delivery bonus, energy cost, boarding rules), so its numbers are not comparable and the harness excludes it by default. Once a model is retrained on the fixed environment, run the harness with --dqn to add its row.

ℹ️Note

Be prepared for the honest outcome: independent Q-learning does not automatically beat SCAN. Multi-agent training is unstable (see the challenges below), and a first training run may plateau far below the heuristics. That gap is itself instructive — closing it is the real exercise.

Reproduce it

One command regenerates the artifact and the table above from a clean checkout:

cd code
pip install -e .
python -m rlbook.examples.evaluate_elevator             # writes code/benchmarks/elevator_baselines.json
python -m rlbook.examples.evaluate_elevator --markdown  # renders the exact table shown above

The run takes a few seconds; the evaluation protocol (seeds, episode count, environment config) lives at the top of the script, and its metadata is embedded in the JSON artifact alongside the results.

What a well-trained agent can learn

The following are coordination strategies that RL dispatch agents can in principle discover, and that the literature reports for related setups (e.g., Crites & Barto, 1996). These are expected/possible behaviors, not measured results from this codebase — verify them in your own runs before claiming them:

🗺️ Implicit Zoning

Elevators could partition floors among themselves (e.g., one covers the lower floors, another the upper floors)

🎯 Anticipatory Positioning

During quiet periods, idle near floors where the traffic pattern makes requests likely

↗️ Direction Awareness

Prefer picking up passengers traveling in the elevator’s current direction

⚖️ Load Balancing

When one elevator is full, others could learn to cover its area

If such behaviors do appear, the interesting part is that they emerge from independent learning against a shared reward — nothing in the code tells elevators to partition floors. Checking whether they emerge in your training run (e.g., by plotting each elevator’s floor-visit histogram) is one of the exercises below.

Challenges & Solutions

❌ Challenge: Slow Initial Learning

With ε=1.0, early episodes are random walks → very negative rewards → slow buffer fill

✓ Solution:

• Pre-fill buffer with nearest-car policy (100 episodes)
• Or use shaped rewards (bonus for approaching requests)

❌ Challenge: Non-Stationarity

All three elevators’ policies change during training → non-stationary environment

✓ Solution:

• Shared replay buffer stabilizes learning
• Target networks reduce moving-target problem
• Slower epsilon decay allows adaptation

❌ Challenge: Exploration in Production

Can’t let elevators explore wildly—real passengers would complain!

✓ Solution:

• Train fully in simulation first
• Deploy with low ε (0.05) for minimal exploration
• Constrain actions to “reasonable” floors only
• Safety fallback: if wait > threshold → nearest-car override

Deployment Considerations

Moving from simulation to real buildings requires careful engineering:

🔄 Sim-to-Real Gap

Gaps:

• Real passengers ≠ Poisson
• Mechanical delays & failures
• Special events (fire, maintenance)

Fixes:

• Use real building data
• Domain randomization
• Continuous fine-tuning

🛡️ Safety Constraints

• Hard timeout: 2 min max wait

• Minimum service frequency/floor

• Emergency mode overrides

⚠️Warning

Implemented as wrappers, not learned!

📊 Monitoring

Track:

• Wait time (mean, p95, max)
• Starvation events
• Utilization & energy

Red flags:

• Wait spike → revert to baseline
• Clustering → check coordination

Extensions

Ways to push this further:

🏢 Larger Buildings

50 floors × 10 elevators. Use graph neural networks (GNNs) to encode relationships. Consider CTDE methods like QMIX.

⚡ Express Elevators

Some elevators skip floors (1, 10, 20…). RL learns when to use express vs local—better than fixed rules.

⚖️ Multi-Objective Optimization

Trade-off wait time vs energy vs fairness. Use weighted reward. Building managers tune weights. Pareto front finds non-dominated policies.

🎯 Destination Dispatch

Passengers enter destination before boarding → full observability → easier credit assignment and better routing.

🔄 Lifelong Learning

Building patterns shift (new tenants, seasons). Keep replay buffer in production, fine-tune nightly, detect distribution drift and retrain automatically.

Try It Yourself

</>Implementation

Hands-On Training

Train your own elevator dispatch agent:

# Clone the repository
git clone https://github.com/ebilgin/rlbook
cd rlbook/code

# Install dependencies
pip install -e .

# Train (wall-clock time depends on your hardware and episode count)
python -m rlbook.examples.train_elevator --episodes 1000

# Try different traffic patterns
python -m rlbook.examples.train_elevator --traffic evening_rush

# Larger building
python -m rlbook.examples.train_elevator --n-floors 20 --n-elevators 5

Or run it in the browser with the notebook:

Run locally with Jupyter

Exercises

Reward Shaping: Modify the reward function to prioritize fairness over average wait time. How does this change behavior?
Architecture Experiments: Try different network sizes ([64,64] vs [256,256]). How does this affect sample efficiency?
Baseline Improvement: Implement a smarter nearest-car policy that considers elevator direction. Can you beat RL?
Traffic Generalization: Train on morning_rush, then evaluate on evening_rush. How well does it transfer?
Communication: Allow elevators to share their target floors with each other. Does this improve coordination?
Emergent Behavior Check: After training, plot each elevator’s floor-visit histogram and idle positions per traffic pattern. Do you see implicit zoning or anticipatory positioning — or does the agent behave no differently from nearest-car?

Key Takeaways

RL shines for coordination problems

When multiple agents work together without explicit communication, RL can discover emergent coordination — but verify it empirically rather than assuming it.

Reward engineering is critical

Small changes (delivery bonus, starvation penalty) drastically affect learned behavior.

Baselines matter

Simple heuristics (SCAN) can be surprisingly good. Always compare to strong baselines, not just random.

Sim-to-real gap is real

Training in simulation is easy; deployment requires careful safety engineering and monitoring.

Multi-agent is hard

Non-stationarity and credit assignment make MARL harder than single-agent. Independent Q-learning with shared replay is a good starting point.

See It In Action

Elevator Dispatch Simulation

Building View

Metrics

Why Traditional Rules Fail

First-Come-First-Served

SCAN Algorithm

Nearest Car

The Problem Domain

Our Test Building

Traffic Patterns

Success Metrics

The MDP Formulation

State Space: What Each Elevator Sees

Own State (15 dims)

Pending Requests (30 dims)

Other Elevators (8 dims)

Time Context (4 dims)

Action Space: Where To Go Next

Reward Design: Balancing Multiple Goals

Episode Structure

The Multi-Agent Challenge

Baseline Approaches

The RL Solution: Independent Q-Learning

Network Architecture

Evaluating the Agent

How to compare policies fairly

Measured baseline results

Reproduce it

What a well-trained agent can learn

Challenges & Solutions

Deployment Considerations

Extensions

Try It Yourself

Hands-On Training

Exercises

Key Takeaways

Further Reading