Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Policy Iteration with Integer Programming for Inventory Management

Pavithra Harsha, Ashish Jagmohan, Jayant Kalagnanam, Brian Quanz, Divya Singhvi

TL;DR

In the simpler setting where optimal replenishment policy is tractable or known near optimal heuristics exist, the RL-based policies can learn near optimal policies, and this improvement in performance of PARL over benchmark algorithms can be directly attributed to better inventory cost management, especially in inventory constrained settings.

Abstract

We present a Reinforcement Learning (RL) based framework for optimizing long-term discounted reward problems with large combinatorial action space and state dependent constraints. These characteristics are common to many operations management problems, e.g., network inventory replenishment, where managers have to deal with uncertain demand, lost sales, and capacity constraints that results in more complex feasible action spaces. Our proposed Programmable Actor Reinforcement Learning (PARL) uses a deep-policy iteration method that leverages neural networks (NNs) to approximate the value function and combines it with mathematical programming (MP) and sample average approximation (SAA) to solve the per-step-action optimally while accounting for combinatorial action spaces and state-dependent constraint sets. We show how the proposed methodology can be applied to complex inventory replenishment problems where analytical solutions are intractable. We also benchmark the proposed algorithm against state-of-the-art RL algorithms and commonly used replenishment heuristics and find it considerably outperforms existing methods by as much as 14.7% on average in various complex supply chain settings. We find that this improvement of PARL over benchmark algorithms can be directly attributed to better inventory cost management, especially in inventory constrained settings. Furthermore, in the simpler setting where optimal replenishment policy is tractable or known near optimal heuristics exist, we find that the RL approaches can learn near optimal policies. Finally, to make RL algorithms more accessible for inventory management researchers, we also discuss the development of a modular Python library that can be used to test the performance of RL algorithms with various supply chain structures and spur future research in developing practical and near-optimal algorithms for inventory management problems.

Deep Policy Iteration with Integer Programming for Inventory Management

TL;DR

In the simpler setting where optimal replenishment policy is tractable or known near optimal heuristics exist, the RL-based policies can learn near optimal policies, and this improvement in performance of PARL over benchmark algorithms can be directly attributed to better inventory cost management, especially in inventory constrained settings.

Abstract

We present a Reinforcement Learning (RL) based framework for optimizing long-term discounted reward problems with large combinatorial action space and state dependent constraints. These characteristics are common to many operations management problems, e.g., network inventory replenishment, where managers have to deal with uncertain demand, lost sales, and capacity constraints that results in more complex feasible action spaces. Our proposed Programmable Actor Reinforcement Learning (PARL) uses a deep-policy iteration method that leverages neural networks (NNs) to approximate the value function and combines it with mathematical programming (MP) and sample average approximation (SAA) to solve the per-step-action optimally while accounting for combinatorial action spaces and state-dependent constraint sets. We show how the proposed methodology can be applied to complex inventory replenishment problems where analytical solutions are intractable. We also benchmark the proposed algorithm against state-of-the-art RL algorithms and commonly used replenishment heuristics and find it considerably outperforms existing methods by as much as 14.7% on average in various complex supply chain settings. We find that this improvement of PARL over benchmark algorithms can be directly attributed to better inventory cost management, especially in inventory constrained settings. Furthermore, in the simpler setting where optimal replenishment policy is tractable or known near optimal heuristics exist, we find that the RL approaches can learn near optimal policies. Finally, to make RL algorithms more accessible for inventory management researchers, we also discuss the development of a modular Python library that can be used to test the performance of RL algorithms with various supply chain structures and spur future research in developing practical and near-optimal algorithms for inventory management problems.
Paper Structure (44 sections, 3 theorems, 19 equations, 12 figures, 8 tables, 1 algorithm)

This paper contains 44 sections, 3 theorems, 19 equations, 12 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Selected Literature Review
  4. Parametric policies for inventory management:
  5. Approximate Dynamic Programming (ADP) and Reinforcement Learning (RL):
  6. RL and Math Programming (MP) for inventory management:
  7. Model and Performance Metrics
  8. Model and notation:
  9. Algorithm
  10. Optimizing over a neural network
  11. Maximizing expected reward with a large action space
  12. Inventory Management Application
  13. Model and System Dynamics
  14. PARL for Inventory Management
  15. Numerical Results and Managerial Insights
  16. ...and 29 more sections

Key Result

Proposition 1

Consider epoch $j$ of the PARL algorithm with a ReLU-network value function estimate $\hat{V}_{\theta}^{\pi_{j-1}}(s)$ for some fixed policy $\pi_{j-1}$. Suppose $\pi_j$ and $\hat{\pi}^{\eta}_j$ are the optimal policies as described in Problem (eq:per_step_problem) and its corresponding SAA approxim

Figures (12)

  • Figure 1: A representative NN that takes as an input, a 6-dimensional state space, and considers a 10 layer NN (including the output layer) with each internal layer containing 9 neurons. Each neuron is defined by weights (W) and bias (b), except for the output layer that is defined with parameter vector $c$ and $b_o$. The output of each neuron uses ReLU activation and passes it as an input to the neurons in the subsequent layer.
  • Figure 2: Example of different multi-echelon supply chain networks. In 1S-3R, a single supplier node serves a set of 3 retail nodes directly. In 1S-2W-3R, the supplier node serves the retail nodes through two warehouses. In 1S-2W-3R (dual sourcing), each retail nodes can is served by two distributors.
  • Figure 3: Learning curves of PARL and benchmark algorithms during training runs.
  • Figure 4: Breakdown of rewards in test across BS, PPO and PARL algorithms. Note that ordering costs include fixed and variable costs of ordering, revenue refers to the revenue earned from sales and reward refers to the revenue net costs incurred (see § \ref{['sec:inventory']} and Table \ref{['table:SCsettings']} for more details).
  • Figure 5: Comparing the optimal ordering policy of PARL and SAC in the 1S$^{\inf}$-1R backorder setting. Here, the optimal static policy is an order-up to policy with 27 units being the order upto level. Observe that both PARL and SAC are able to learn the optimal order-up-to policy across various on-hand inventory states observed in test (99.9% of the time). Higher variance in comparison to SAC for PARL can be attributed to its deterministic policy structure that optimizes over the learned critic with a four dimensional input state.
  • ...and 7 more figures

Theorems & Definitions (3)

  • Proposition 1
  • Theorem 1: Theorem 5.3 of shapiro2014lectures
  • Proposition 2: Proposition 7 of shapiro2003monte