The observation for elevator $i$ at time $t$ is:

$\mathbf{o}_i^t = [\mathbf{s}_i^t, \mathbf{r}^t, \mathbf{e}_{-i}^t, \mathbf{c}^t]$

where:

• $\mathbf{s}_i^t \in \mathbb{R}^{14}$: Own state (floor, direction one-hot, passenger count, destination floors binary)
• $\mathbf{r}^t \in \mathbb{R}^{20}$: Requests (waiting passengers per floor, up/down buttons)
• $\mathbf{e}_{-i}^t \in \mathbb{R}^{8}$: Other elevators (2 elevators × 4 features each)
• $\mathbf{c}^t \in \mathbb{R}^{4}$: Traffic context (one-hot)

Total observation dimension: $d = 46$

At each timestep $t$, each elevator $i$ selects an action:

$a_i^t \in \mathcal{A}_i = \{0, 1, \ldots, 9\}$

The joint action is $\mathbf{a}^t = (a_1^t, a_2^t, a_3^t)$.

Formally, the reward at timestep $t$ is:

$r^t = -\sum_{p \in W^t} w(p) + 10 \cdot |D^t| - 0.1 \cdot \sum_{i=1}^{3} m_i^t$

where:

• $W^t$: set of waiting passengers at time $t$
• $w(p) = \begin{cases} 6 & \text{if wait time} > 60 \\ 1 & \text{otherwise} \end{cases}$
• $D^t$: set of passengers delivered at time $t$
• $m_i^t$: floors moved by elevator $i$ at time $t$

The discount factor is $\gamma = 0.99$ (episodes are short, so we weight near-future heavily).

This is a Decentralized Partially Observable Markov Decision Process (Dec-POMDP).

Formally:

• Agents: $\mathcal{N} = \{1, 2, 3\}$ (three elevators)
• Joint observation: $\mathbf{o}^t = (o_1^t, o_2^t, o_3^t)$ where each $o_i^t$ depends on true state $s^t$
• Joint action: $\mathbf{a}^t = (a_1^t, a_2^t, a_3^t)$
• Shared reward: $r^t$ (all agents receive the same reward signal)
• Transition: $s^{t+1} \sim P(\cdot | s^t, \mathbf{a}^t)$

The goal is to find a joint policy $\pi = (\pi_1, \pi_2, \pi_3)$ that maximizes expected return:

$J(\pi) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^{T} \gamma^t r^t\right]$

Each elevator $i$ learns a Q-function $Q_i: \mathcal{O}_i \times \mathcal{A}_i \to \mathbb{R}$ via:

$Q_i(o_i^t, a_i^t) \leftarrow Q_i(o_i^t, a_i^t) + \alpha \left[r^t + \gamma \max_{a_i'} Q_i(o_i^{t+1}, a_i') - Q_i(o_i^t, a_i^t)\right]$

Note that the reward $r^t$ is shared (global), but each elevator updates based on its own observation-action pairs.

The policy for elevator $i$ is:

$\pi_i(o_i) = \arg\max_{a_i} Q_i(o_i, a_i)$

During training, we use ε-greedy exploration:

$a_i^t = \text{argmax}_{a_i} Q_i(o_i^t, a_i) \text{ with prob. } 1-\epsilon, \text{ else random}$

Or more precisely:

• With probability $\epsilon$: select random floor
• With probability $1-\epsilon$: select $\pi_i(o_i^t)$