Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy-Based Deep Reinforcement Learning Hyperheuristics for Job-Shop Scheduling Problems

Sofiene Lassoued, Asrat Gobachew, Stefan Lier, Andreas Schwung

TL;DR

This work tackles the Job Shop Scheduling Problem (JSSP) by introducing a policy-based deep reinforcement learning hyper-heuristic (DRL-HH) that operates on a colored timed Petri-net environment. Action masking via Petri-net guards ensures only feasible low-level heuristics are evaluated, while a commitment mechanism temporally extends heuristic choices to improve credit assignment and training stability. The high-level policy selects among a fixed set of dispatching rules, enabling state-aware, interpretable scheduling decisions that generalize across problem sizes. Empirical results on Taillard benchmarks show the approach outperforms classic heuristics, metaheuristics, and several neural-network-based methods, with 5-step commitment providing the best trade-off between adaptability and stability. Overall, the combination of action prefiltering and temporal abstraction yields a scalable, explainable DRL-HH framework for complex manufacturing scheduling.

Abstract

This paper proposes a policy-based deep reinforcement learning hyper-heuristic framework for solving the Job Shop Scheduling Problem. The hyper-heuristic agent learns to switch scheduling rules based on the system state dynamically. We extend the hyper-heuristic framework with two key mechanisms. First, action prefiltering restricts decision-making to feasible low-level actions, enabling low-level heuristics to be evaluated independently of environmental constraints and providing an unbiased assessment. Second, a commitment mechanism regulates the frequency of heuristic switching. We investigate the impact of different commitment strategies, from step-wise switching to full-episode commitment, on both training behavior and makespan. Additionally, we compare two action selection strategies at the policy level: deterministic greedy selection and stochastic sampling. Computational experiments on standard JSSP benchmarks demonstrate that the proposed approach outperforms traditional heuristics, metaheuristics, and recent neural network-based scheduling methods

Policy-Based Deep Reinforcement Learning Hyperheuristics for Job-Shop Scheduling Problems

TL;DR

This work tackles the Job Shop Scheduling Problem (JSSP) by introducing a policy-based deep reinforcement learning hyper-heuristic (DRL-HH) that operates on a colored timed Petri-net environment. Action masking via Petri-net guards ensures only feasible low-level heuristics are evaluated, while a commitment mechanism temporally extends heuristic choices to improve credit assignment and training stability. The high-level policy selects among a fixed set of dispatching rules, enabling state-aware, interpretable scheduling decisions that generalize across problem sizes. Empirical results on Taillard benchmarks show the approach outperforms classic heuristics, metaheuristics, and several neural-network-based methods, with 5-step commitment providing the best trade-off between adaptability and stability. Overall, the combination of action prefiltering and temporal abstraction yields a scalable, explainable DRL-HH framework for complex manufacturing scheduling.

Abstract

This paper proposes a policy-based deep reinforcement learning hyper-heuristic framework for solving the Job Shop Scheduling Problem. The hyper-heuristic agent learns to switch scheduling rules based on the system state dynamically. We extend the hyper-heuristic framework with two key mechanisms. First, action prefiltering restricts decision-making to feasible low-level actions, enabling low-level heuristics to be evaluated independently of environmental constraints and providing an unbiased assessment. Second, a commitment mechanism regulates the frequency of heuristic switching. We investigate the impact of different commitment strategies, from step-wise switching to full-episode commitment, on both training behavior and makespan. Additionally, we compare two action selection strategies at the policy level: deterministic greedy selection and stochastic sampling. Computational experiments on standard JSSP benchmarks demonstrate that the proposed approach outperforms traditional heuristics, metaheuristics, and recent neural network-based scheduling methods
Paper Structure (24 sections, 20 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 24 sections, 20 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Job shop scheduling problem definition
  5. Colored timed Petrinets
  6. Reinforcement learning framework
  7. Proximal Policy Optimization algorithm
  8. The PetriRL Hyper-Heuristic Framework
  9. The JSSP Environment
  10. PetriRL Hyper Heuristic overview
  11. Analysis of the approach's features
  12. Reduced and fixed Action Space size
  13. Policy Search over Heuristics
  14. State-Dependent Switching
  15. Rule commitment and effect Credit Assignment
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Representation of a 5-job x 4-machine job shop scheduling problem modeled using a timed colored Petri net.
  • Figure 2: PetriRL Hyper-heuristic framework: The policy network (left) selects an appropriate heuristic based on the current state. The chosen heuristic then decides the next action, which is executed in the colored timed Petri net environment modeling the JSSP (right).
  • Figure 3: Agent training performances on a 20 jobs x 20 machines instance . (a) the episode mean reward , (b) the combined training loss,(c) the entropy loss (d) the clipping range. The agent is trained for 1e6 steps with 5-step commitment using the maskable PPO algorithm.
  • Figure 4: Comparison of two control strategies applied to the same environment and problem instance (20 jobs x 20 machines): a hyperheuristic agent selecting among a set of heuristic rules (blue) and a reinforcement learning agent acting directly on the environment's action space (red). Subfigures show: (a) rewards collected by the hyperheuristic agent, (b) its associated training loss function, (c) rewards collected by the direct-action RL agent, and (d) its corresponding training loss function.