Zero-shot Generalization in Inventory Management: Train, then Estimate and Decide

TL;DR

This paper introduces TED, a Train–Estimate–Decide framework, underpinned by a Super-MDP, to enable zero-shot generalization of generally capable agents (GCAs) in inventory management under parameter uncertainty. A GC-LSN policy is trained offline to operate across a broad class of periodic-review lost-sales problems, and is deployed online using real-time parameter estimates (e.g., via Kaplan–Meier) without retraining. The authors establish a Lipschitz Super-MDP framework and provide a theoretical bound on performance loss due to parameter estimation error, showing that consistency of the estimator drives the GCA's performance to that of a clairvoyant policy. Empirically, GC-LSN outperforms traditional policies when parameters are known, and GC-LSN-E demonstrates strong empirical performance against online-learning baselines under censored and unknown demand/lead-time distributions, highlighting practical relevance for data-scarce, dynamic inventory settings. The work contributes a unifying framework, demonstrates strong empirical results, and offers open-source tools to advance practical deployment of DRL in inventory systems.

Abstract

Deploying deep reinforcement learning (DRL) in real-world inventory management presents challenges, including dynamic environments and uncertain problem parameters, e.g. demand and lead time distributions. These challenges highlight a research gap, suggesting a need for a unifying framework to model and solve sequential decision-making under parameter uncertainty. We address this by exploring an underexplored area of DRL for inventory management: training generally capable agents (GCAs) under zero-shot generalization (ZSG). Here, GCAs are advanced DRL policies designed to handle a broad range of sampled problem instances with diverse inventory challenges. ZSG refers to the ability to successfully apply learned policies to unseen instances with unknown parameters without retraining. We propose a unifying Super-Markov Decision Process formulation and the Train, then Estimate and Decide (TED) framework to train and deploy a GCA tailored to inventory management applications. The TED framework consists of three phases: training a GCA on varied problem instances, continuously estimating problem parameters during deployment, and making decisions based on these estimates. Applied to periodic review inventory problems with lost sales, cyclic demand patterns, and stochastic lead times, our trained agent, the Generally Capable Lost Sales Network (GC-LSN) consistently outperforms well-known traditional policies when problem parameters are known. Moreover, under conditions where demand and/or lead time distributions are initially unknown and must be estimated, we benchmark against online learning methods that provide worst-case performance guarantees. Our GC-LSN policy, paired with the Kaplan-Meier estimator, is demonstrated to complement these methods by providing superior empirical performance.

Figures (5)

• Figure 1: Train, then Estimate and Decide framework for solving sequential decision-making problems with dynamic parameter estimation. In each decision period, if parameters are unknown, the TED framework first estimates the parameterization (Estimate) from the observations and then determines the action (Decide) using the trained GCA. If parameters are already known, the GCA can be applied directly without the Estimate step.
• Figure 2: Construction of $\hat{\mathcal{P}}$ and $\hat{\mathcal{H}}$. As more parameterizations are sampled from $\hat{\mathcal{H}}$, the training set covers $\hat{\mathcal{P}}$ more densely; since $\mathcal{P}\subseteq\hat{\mathcal{P}}$, this increases the likelihood that unseen test instances lie near trained ones, improving ZSG.
• Figure 3: Left: Order quantities placed by GC-LSN, BSP, and C-BSP for a fixed state (on hand inventory and each pipeline have 8 units) when the penalty cost $p$ increases. Demand ($K=1$) is geometric distributed with mean $8$ and lead time is 8. Right: Order quantities taken by GC-LSN-E when supplied with estimated demand distributions. Demand ($K=1$) is geometric distributed with mean $8$, lead time is 8, $p=69.0$.
• Figure 4: Relative cost gap—lower is better; % omitted—between GC-LSN and C-BSP for a Case 1 parametrization ($K=1$, geometric distributed demand, lead time is 8) as the mean demand $\mu$ and penalty cost $p$ move beyond the training bounds $\mu_{\max}=12.0$ and $p_{\max}=100.0$.
• Figure 5: Two-dimensional representation of the probable parameter space $\hat{\mathcal{P}}$ for Case 1 instances with deterministic lead time of $6$ and geometrically distributed demand.

Theorems & Definitions (16)

• Definition 1: Markov Decision Process
• Definition 2: Parameterization and Parameter Space
• Definition 3: Super-Markov Decision Process
• Remark 1
• Definition 4: Lipschitz Super-Markov Decision Processes
• Definition 5: Parameterization Distance
• Definition 6: Consistent Estimator
• Theorem 1: Bound on Cost Difference Due to Estimation Error
• proof
• proof