Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chains

Francesco Stranieri, Fabio Stella

TL;DR

This work tackles inventory management in a two-echelon supply chain under seasonal and stochastic demand by applying deep reinforcement learning. It introduces a stochastic, divergent two-echelon SCIM environment with $I$ product types and $J$ warehouses, a continuous action space, zero lead times, and an open-source library for reproducible benchmarking. The study compares A3C, PPO, and VPG against a Bayesian-optimized ($s$, $Q$) policy and an oracle across three scenarios (1P1W, 1P3W, 2P2W), finding that PPO generally offers the most robust performance while BO excels in simpler spaces. The results support the practical viability of DRL for complex inventory problems and provide a reusable benchmarking platform for future research and industry validation, enabling fair comparisons across topologies and configurations.

Abstract

In this study, we analyze and compare the performance of state-of-the-art deep reinforcement learning algorithms for solving the supply chain inventory management problem. This complex sequential decision-making problem consists of determining the optimal quantity of products to be produced and shipped across different warehouses over a given time horizon. In particular, we present a mathematical formulation of a two-echelon supply chain environment with stochastic and seasonal demand, which allows managing an arbitrary number of warehouses and product types. Through a rich set of numerical experiments, we compare the performance of different deep reinforcement learning algorithms under various supply chain structures, topologies, demands, capacities, and costs. The results of the experimental plan indicate that deep reinforcement learning algorithms outperform traditional inventory management strategies, such as the static (s, Q)-policy. Furthermore, this study provides detailed insight into the design and development of an open-source software library that provides a customizable environment for solving the supply chain inventory management problem using a wide range of data-driven approaches.

Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chains

TL;DR

This work tackles inventory management in a two-echelon supply chain under seasonal and stochastic demand by applying deep reinforcement learning. It introduces a stochastic, divergent two-echelon SCIM environment with product types and warehouses, a continuous action space, zero lead times, and an open-source library for reproducible benchmarking. The study compares A3C, PPO, and VPG against a Bayesian-optimized (, ) policy and an oracle across three scenarios (1P1W, 1P3W, 2P2W), finding that PPO generally offers the most robust performance while BO excels in simpler spaces. The results support the practical viability of DRL for complex inventory problems and provide a reusable benchmarking platform for future research and industry validation, enabling fair comparisons across topologies and configurations.

Abstract

In this study, we analyze and compare the performance of state-of-the-art deep reinforcement learning algorithms for solving the supply chain inventory management problem. This complex sequential decision-making problem consists of determining the optimal quantity of products to be produced and shipped across different warehouses over a given time horizon. In particular, we present a mathematical formulation of a two-echelon supply chain environment with stochastic and seasonal demand, which allows managing an arbitrary number of warehouses and product types. Through a rich set of numerical experiments, we compare the performance of different deep reinforcement learning algorithms under various supply chain structures, topologies, demands, capacities, and costs. The results of the experimental plan indicate that deep reinforcement learning algorithms outperform traditional inventory management strategies, such as the static (s, Q)-policy. Furthermore, this study provides detailed insight into the design and development of an open-source software library that provides a customizable environment for solving the supply chain inventory management problem using a wide range of data-driven approaches.
Paper Structure (9 sections, 5 equations, 3 figures, 1 table)

This paper contains 9 sections, 5 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Problem Definition
  4. Environment Formulation
  5. Numerical Experiments
  6. Scenarios Considered
  7. Results
  8. Discussions and Conclusions
  9. Future Research

Figures (3)

  • Figure 1: Agent-environment interface in an MDP (taken from sutton2018reinforcement).
  • Figure 2: A divergent two-echelon supply chain consisting of a factory and its warehouse (first echelon), plus three distribution warehouses (second echelon). Shopping carts represent customers' demands.
  • Figure 3: Some instances of different demands behavior generated according to \ref{['eq:demand']} for different topologies and configurations of the SCIM problem: \ref{['fig:1p1wd']} one product type and one distribution warehouse with $d_{max} = 5$ and $d_{var} = 3$; \ref{['fig:1p3wd']} one product type and three distribution warehouses with $d_{max} = 7$ and $d_{var} = 2$; and \ref{['fig:2p2wd']} two product types and two distribution warehouses with $d_{max} = (3, 6)$ and $d_{var} = (2, 1)$ (referring to the values in round brackets, the first denotes the first product type, whereas the second indicates the second product type).