A Curriculum-Based Deep Reinforcement Learning Framework for the Electric Vehicle Routing Problem

TL;DR

The paper tackles the Electric Vehicle Routing Problem with Time Windows (EVRPTW), a highly constrained, NP-hard optimization task where traditional solvers struggle for real-time applications. It proposes a curriculum-based deep reinforcement learning framework (CB-DRL) that uses a three-phase constraint curriculum and a modified proximal policy optimization algorithm, supported by a heterogeneous graph attention encoder. Key contributions include decoupling learning into topology, energy management, and scheduling phases, achieving robust zero-shot generalization from $N=10$ to unseen sizes up to $N=100$, and delivering competitive or superior solution quality with higher feasibility on out-of-distribution instances. The framework demonstrates scalable, reliable performance that narrows the gap between neural speed and operational reliability, and suggests future work in stochastic environments and integration with meta-heuristics for even larger logistics networks.

Abstract

The electric vehicle routing problem with time windows (EVRPTW) is a complex optimization problem in sustainable logistics, where routing decisions must minimize total travel distance, fleet size, and battery usage while satisfying strict customer time constraints. Although deep reinforcement learning (DRL) has shown great potential as an alternative to classical heuristics and exact solvers, existing DRL models often struggle to maintain training stability-failing to converge or generalize when constraints are dense. In this study, we propose a curriculum-based deep reinforcement learning (CB-DRL) framework designed to resolve this instability. The framework utilizes a structured three-phase curriculum that gradually increases problem complexity: the agent first learns distance and fleet optimization (Phase A), then battery management (Phase B), and finally the full EVRPTW (Phase C). To ensure stable learning across phases, the framework employs a modified proximal policy optimization algorithm with phase-specific hyperparameters, value and advantage clipping, and adaptive learning-rate scheduling. The policy network is built upon a heterogeneous graph attention encoder enhanced by global-local attention and feature-wise linear modulation. This specialized architecture explicitly captures the distinct properties of depots, customers, and charging stations. Trained exclusively on small instances with N=10 customers, the model demonstrates robust generalization to unseen instances ranging from N=5 to N=100, significantly outperforming standard baselines on medium-scale problems. Experimental results confirm that this curriculum-guided approach achieves high feasibility rates and competitive solution quality on out-of-distribution instances where standard DRL baselines fail, effectively bridging the gap between neural speed and operational reliability.

Figures (3)

• Figure 1: Overview of the proposed curriculum-based deep reinforcement learning (CB-DRL) framework.
• Figure 2: The three-phase constraint curriculum.
• Figure 3: Performance metrics (A, B) and routing behavior (C, D) comparing standard PPO and CB-DRL.