Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic resource matching in manufacturing using deep reinforcement learning

Saunak Kumar Panda, Yisha Xiang, Ruiqi Liu

Abstract

Matching plays an important role in the logical allocation of resources across a wide range of industries. The benefits of matching have been increasingly recognized in manufacturing industries. In particular, capacity sharing has received much attention recently. In this paper, we consider the problem of dynamically matching demand-capacity types of manufacturing resources. We formulate the multi-period, many-to-many manufacturing resource-matching problem as a sequential decision process. The formulated manufacturing resource-matching problem involves large state and action spaces, and it is not practical to accurately model the joint distribution of various types of demands. To address the curse of dimensionality and the difficulty of explicitly modeling the transition dynamics, we use a model-free deep reinforcement learning approach to find optimal matching policies. Moreover, to tackle the issue of infeasible actions and slow convergence due to initial biased estimates caused by the maximum operator in Q-learning, we introduce two penalties to the traditional Q-learning algorithm: a domain knowledge-based penalty based on a prior policy and an infeasibility penalty that conforms to the demand-supply constraints. We establish theoretical results on the convergence of our domain knowledge-informed Q-learning providing performance guarantee for small-size problems. For large-size problems, we further inject our modified approach into the deep deterministic policy gradient (DDPG) algorithm, which we refer to as domain knowledge-informed DDPG (DKDDPG). In our computational study, including small- and large-scale experiments, DKDDPG consistently outperformed traditional DDPG and other RL algorithms, yielding higher rewards and demonstrating greater efficiency in time and episodes.

Dynamic resource matching in manufacturing using deep reinforcement learning

Abstract

Matching plays an important role in the logical allocation of resources across a wide range of industries. The benefits of matching have been increasingly recognized in manufacturing industries. In particular, capacity sharing has received much attention recently. In this paper, we consider the problem of dynamically matching demand-capacity types of manufacturing resources. We formulate the multi-period, many-to-many manufacturing resource-matching problem as a sequential decision process. The formulated manufacturing resource-matching problem involves large state and action spaces, and it is not practical to accurately model the joint distribution of various types of demands. To address the curse of dimensionality and the difficulty of explicitly modeling the transition dynamics, we use a model-free deep reinforcement learning approach to find optimal matching policies. Moreover, to tackle the issue of infeasible actions and slow convergence due to initial biased estimates caused by the maximum operator in Q-learning, we introduce two penalties to the traditional Q-learning algorithm: a domain knowledge-based penalty based on a prior policy and an infeasibility penalty that conforms to the demand-supply constraints. We establish theoretical results on the convergence of our domain knowledge-informed Q-learning providing performance guarantee for small-size problems. For large-size problems, we further inject our modified approach into the deep deterministic policy gradient (DDPG) algorithm, which we refer to as domain knowledge-informed DDPG (DKDDPG). In our computational study, including small- and large-scale experiments, DKDDPG consistently outperformed traditional DDPG and other RL algorithms, yielding higher rewards and demonstrating greater efficiency in time and episodes.

Paper Structure

This paper contains 23 sections, 6 theorems, 32 equations, 8 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Matching Problem
  4. Reinforcement learning
  5. RL for Combinatorial optimization
  6. Problem Setting
  7. MDP framework
  8. Domain knowledge-informed Q-learning
  9. Prior policy $\mu$
  10. Regularization parameter $\beta$
  11. Infeasibility penalty
  12. Convergence of domain knowledge-informed Q-learning
  13. Value-penalty function with fixed $\beta$
  14. Value-penalty function with changing $\beta_t$
  15. Domain knowledge-informed Deep Deterministic Policy gradient (DKDDPG) Algorithm
  16. ...and 8 more sections

Key Result

Theorem 1

Let $Q_t$ be the sequence generated by the iteration eq:newQ. We assume that the step-sizes $\alpha_t$ are non-negative and satisfy Then $Q_t$ converges to $Q^*$ with probability 1.

Figures (8)

  • Figure 1: Material (product) upgrade application of Horizontal differentiation
  • Figure 2: Dynamic matching problem illustration
  • Figure 3: Neural network architecture of DKDDPG
  • Figure 4: Action transformation for a 2x2 matching example
  • Figure 5: 2x2 Matching problem
  • ...and 3 more figures

Theorems & Definitions (8)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Theorem 2
  • proof
  • Lemma 2
  • Theorem 2
  • proof