Cluster-Aware Attention-Based Deep Reinforcement Learning for Pickup and Delivery Problems

TL;DR

This paper proposes CAADRL (Cluster-Aware Attention-based Deep Reinforcement Learning), a DRL framework that explicitly exploits the multi-scale structure of PDP instances via cluster-aware encoding and hierarchical decoding and achieves results with substantially lower inference time than neural collaborative-search baselines.

Abstract

The Pickup and Delivery Problem (PDP) is a fundamental and challenging variant of the Vehicle Routing Problem, characterized by tightly coupled pickup--delivery pairs, precedence constraints, and spatial layouts that often exhibit clustering. Existing deep reinforcement learning (DRL) approaches either model all nodes on a flat graph, relying on implicit learning to enforce constraints, or achieve strong performance through inference-time collaborative search at the cost of substantial latency. In this paper, we propose \emph{CAADRL} (Cluster-Aware Attention-based Deep Reinforcement Learning), a DRL framework that explicitly exploits the multi-scale structure of PDP instances via cluster-aware encoding and hierarchical decoding. The encoder builds on a Transformer and combines global self-attention with intra-cluster attention over depot, pickup, and delivery nodes, producing embeddings that are both globally informative and locally role-aware. Based on these embeddings, we introduce a Dynamic Dual-Decoder with a learnable gate that balances intra-cluster routing and inter-cluster transitions at each step. The policy is trained end-to-end with a POMO-style policy gradient scheme using multiple symmetric rollouts per instance. Experiments on synthetic clustered and uniform PDP benchmarks show that CAADRL matches or improves upon strong state-of-the-art baselines on clustered instances and remains highly competitive on uniform instances, particularly as problem size increases. Crucially, our method achieves these results with substantially lower inference time than neural collaborative-search baselines, suggesting that explicitly modeling cluster structure provides an effective and efficient inductive bias for neural PDP solvers.

Figures (5)

• Figure 1: Illustration of the sequential decision-making process for PDP modeled as an MDP. The process evolves from $t=0$ to $t=2$, showing the transition of states $s_t$ based on selected actions $A_t$ (visiting nodes). The dashed orange ovals highlight the spatial cluster structure (pickup and delivery regions) explicitly considered in our framework. The reward $r_{t+1}$ corresponds to the negative distance traveled.
• Figure 2: Cluster-Aware Policy Network architecture.
• Figure 3: Training convergence on clustered instances.
• Figure 4: Training convergence on uniform instances.
• Figure 5: Cross-size generalization of CAADRL under greedy decoding. Solid bars indicate uniform instances and hatched bars indicate clustered instances. Colors denote the training problem size (PDP10/20/40/80). Error bars show the standard deviation over 100 test instances.