ORPR: An OR-Guided Pretrain-then-Reinforce Learning Model for Inventory Management
Lingjie Zhao, Xue Yu, Yongzhi Qi, Hao Hu, Jianshen Zhang, Yingzheng Ma, Shuyu Han, Wei Qi, Zuo-Jun Max Shen
TL;DR
This paper introduces an OR-guided Pretrain-then-Reinforce framework that tightly couples a simulation-augmented OR model with a domain-informed DL foundation and reinforcement learning fine-tuning to achieve scalable, high-performance multi-category replenishment. By generating high-quality OR-based reference decisions and using them to pretrain a compact neural network, followed by RL alignment that blends rule-based and simulation-based rewards, the approach achieves strong generalization and rapid adaptation to promotions. Empirical validation includes extensive offline experiments on JD.com data and a 30-day field deployment demonstrating significant gains in turnover reduction, in-stock rates, and holding-cost savings, with a Difference-in-Differences analysis supporting causal impact. The work argues for a lightweight, domain-informed AI that leverages structured OR logic rather than relying on brute-force model scaling, offering a practical blueprint for scalable intelligent supply chain management.
Abstract
As the pursuit of synergy between Artificial Intelligence (AI) and Operations Research (OR) gains momentum in handling complex inventory systems, a critical challenge persists: how to effectively reconcile AI's adaptive perception with OR's structural rigor. To bridge this gap, we propose a novel OR-Guided "Pretrain-then-Reinforce" framework. To provide structured guidance, we propose a simulation-augmented OR model that generates high-quality reference decisions, implicitly capturing complex business constraints and managerial preferences. Leveraging these OR-derived decisions as foundational training labels, we design a domain-informed deep learning foundation model to establish foundational decision-making capabilities, followed by a reinforcement learning (RL) fine-tuning stage. Uniquely, we position RL as a deep alignment mechanism that enables the AI agent to internalize the optimality principles of OR, while simultaneously leveraging exploration for general policy refinement and allowing expert guidance for scenario-specific adaptation (e.g., promotional events). Validated through extensive numerical experiments and a field deployment at JD.com augmented by a Difference-in-Differences (DiD) analysis, our model significantly outperforms incumbent industrial practices, delivering real-world gains of a 5.27-day reduction in turnover and a 2.29% increase in in-stock rates, alongside a 29.95% decrease in holding costs. Contrary to the prevailing trend of brute-force model scaling, our study demonstrates that a lightweight, domain-informed model can deliver state-of-the-art performance and robust transferability when guided by structured OR logic. This approach offers a scalable and cost-effective paradigm for intelligent supply chain management, highlighting the value of deeply aligning AI with OR.