Policy-Based Reinforcement Learning with Action Masking for Dynamic Job Shop Scheduling under Uncertainty: Handling Random Arrivals and Machine Failures

TL;DR

This paper tackles Dynamic Job Shop Scheduling under uncertainty by blending a model-based Colored-Timed Petri Net environment with a Maskable PPO agent to restrict decisions to feasible actions. It explicitly models random job arrivals with a Gamma distribution $\text{Gamma}(\alpha,\beta)$ and machine failures with a Weibull distribution (shape $\beta$, scale $\eta$), enabling realistic disruption scenarios. Two action-masking strategies are explored: a gradient-free mask that zeros invalid actions and a gradient-based penalty that learns feasibility constraints via $L_{inv}(\theta)=\lambda_{inv}\sum_{a \in \mathcal{A}_{inv}(s)} \pi_\theta(a|s)$. Extensive experiments on Raj and Taillard benchmarks show that PetriRL consistently outperforms dispatching-rule baselines, with notable improvements in makespan and robustness, and the authors provide an open-source PyPI package for reproducible research. Overall, the work demonstrates how integrating interpretable Petri-net models with adaptive, mask-aware RL yields a scalable, disruption-resilient scheduling framework for real-time manufacturing.

Abstract

We present a novel framework for solving Dynamic Job Shop Scheduling Problems under uncertainty, addressing the challenges introduced by stochastic job arrivals and unexpected machine breakdowns. Our approach follows a model-based paradigm, using Coloured Timed Petri Nets to represent the scheduling environment, and Maskable Proximal Policy Optimization to enable dynamic decision-making while restricting the agent to feasible actions at each decision point. To simulate realistic industrial conditions, dynamic job arrivals are modeled using a Gamma distribution, which captures complex temporal patterns such as bursts, clustering, and fluctuating workloads. Machine failures are modeled using a Weibull distribution to represent age-dependent degradation and wear-out dynamics. These stochastic models enable the framework to reflect real-world manufacturing scenarios better. In addition, we study two action-masking strategies: a non-gradient approach that overrides the probabilities of invalid actions, and a gradient-based approach that assigns negative gradients to invalid actions within the policy network. We conduct extensive experiments on dynamic JSSP benchmarks, demonstrating that our method consistently outperforms traditional heuristic and rule-based approaches in terms of makespan minimization. The results highlight the strength of combining interpretable Petri-net-based models with adaptive reinforcement learning policies, yielding a resilient, scalable, and explainable framework for real-time scheduling in dynamic and uncertain manufacturing environments.

Figures (11)

• Figure 1: A taxonomy map of dynamic scheduling, highlighting the main branches, approaches, strategies, policies, and methods Wang.2020.
• Figure 2: Examples of Petri Net Variants: (a) Basic Petri Net, (b) Colored Petri Net, and (c) Colored-Timed Petri Net.
• Figure 3: Illustration of the synergistic interaction between Petri net guard functions and the action masking mechanism in the policy network.
• Figure 4: Colored Timed Petri Net modeling a Job Shop Scheduling Problem with 3 jobs and 3 machines.
• Figure 5: Top: Probability density functions of the Weibull distribution for different shape parameters $\beta$. The shape $\beta = 0.5$ models infant mortality $\beta = 1$ models random failure with a constant rate, and $\beta = 5$ models wear-out failure. Bottom: The corresponding hazard functions $h(t)$, which illustrate how the failure rate evolves depending on the value of $\beta$.