Iterative Multi-Agent Reinforcement Learning: A Novel Approach Toward Real-World Multi-Echelon Inventory Optimization

TL;DR

This work tackles the complexity of multi-echelon inventory optimization (MEIO) by evaluating deep reinforcement learning (DRL) approaches and introducing Iterative Multi-Agent Reinforcement Learning (IMARL) to mitigate the curse of dimensionality and non-stationarity. Building on a replication of a state-of-the-art DRL model, the authors extend the framework with Graph Neural Networks and multi-agent variants, ultimately proposing IMARL, which trains agents sequentially to achieve stable, scalable policies. Across a complexity grid of 13 supply-chain scenarios, IMARL consistently outperforms a generalized decomposition-aggregation (DA) heuristic, delivering $6 ext{%}$ to $14 ext{%}$ cost savings on average and demonstrating superior reliability. The findings suggest DRL, particularly IMARL, can address real-world MEIO challenges and motivate further research into transfer learning, non-stationary environments, and efficiency improvements for practical deployment and environmental benefits in supply chains.

Abstract

Multi-echelon inventory optimization (MEIO) is critical for effective supply chain management, but its inherent complexity can pose significant challenges. Heuristics are commonly used to address this complexity, yet they often face limitations in scope and scalability. Recent research has found deep reinforcement learning (DRL) to be a promising alternative to traditional heuristics, offering greater versatility by utilizing dynamic decision-making capabilities. However, since DRL is known to struggle with the curse of dimensionality, its relevance to complex real-life supply chain scenarios is still to be determined. This thesis investigates DRL's applicability to MEIO problems of increasing complexity. A state-of-the-art DRL model was replicated, enhanced, and tested across 13 supply chain scenarios, combining diverse network structures and parameters. To address DRL's challenges with dimensionality, additional models leveraging graph neural networks (GNNs) and multi-agent reinforcement learning (MARL) were developed, culminating in the novel iterative multi-agent reinforcement learning (IMARL) approach. IMARL demonstrated superior scalability, effectiveness, and reliability in optimizing inventory policies, consistently outperforming benchmarks. These findings confirm the potential of DRL, particularly IMARL, to address real-world supply chain challenges and call for additional research to further expand its applicability.

Figures (10)

• Figure 1: Three Types of Inventory Network: Example structures for the primary types of inventory networks: serial, divergent, and general.
• Figure 2: Modeled Network Structures Visual depiction of the four network structures used in the experiments, differentiated by the number of stock points and echelons, as well as the type of inventory network (divergent or general).
• Figure 3: Real-Life Demand and Shipment Lead Time Density Plots Probability density plots visualizing the variability in real-life demand and shipment lead times. The plots represent the first 18 products in the dataset (Yang et al., 2023b), corresponding to the maximum number of stock points modeled in the network structures.
• Figure 4: Complexity Grid Framework A structured framework mapping the complexity of supply chain scenarios by network structure and parameters, highlighting the scenarios included in this study and those covered in previous research by Geevers et al. (2024).
• Figure 5: Best Experiment Results per Model and Complexity Scenario A heat map showing the best results achieved by each tested model across the 13 supply chain scenarios. Cell colors correspond to percentage values provided in parentheses after the absolute results, reflecting cost savings relative to the benchmark.