Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Flexible Manufacturing Systems Intralogistics: Dynamic Optimization of AGVs and Tool Sharing Using Coloured-Timed Petri Nets and Actor-Critic RL with Actions Masking

Sofiene Lassoued, Laxmikant Shrikant Bahetic, Nathalie Weiß-Borkowskib, Stefan Lierc, Andreas Schwunga

TL;DR

This work tackles the complex scheduling of flexible manufacturing systems (FMS) with constrained material handling via automated guided vehicles (AGVs) and tool sharing. It proposes a hybrid framework that combines Colored-Timed Petri Nets (CTPNs) with an actor-critic reinforcement learning approach using action masking (Maskable PPO) to manage the expanded action space. Key contributions include a modular PetriNet-based model, a gym-compatible large-scale Taillard-inspired benchmark, and a thorough ablation study confirming the roles of reward shaping, lookahead, and masking. Results show that the proposed PetriRL method matches traditional methods on small instances and substantially outperforms them on large-scale problems, with dramatic reductions in computation time and strong generalization across varied instance sizes, enabling dynamic adaptation to late orders and disturbances.

Abstract

Flexible Manufacturing Systems (FMS) are pivotal in optimizing production processes in today's rapidly evolving manufacturing landscape. This paper advances the traditional job shop scheduling problem by incorporating additional complexities through the simultaneous integration of automated guided vehicles (AGVs) and tool-sharing systems. We propose a novel approach that combines Colored-Timed Petri Nets (CTPNs) with actor-critic model-based reinforcement learning (MBRL), effectively addressing the multifaceted challenges associated with FMS. CTPNs provide a formal modeling structure and dynamic action masking, significantly reducing the action search space, while MBRL ensures adaptability to changing environments through the learned policy. Leveraging the advantages of MBRL, we incorporate a lookahead strategy for optimal positioning of AGVs, improving operational efficiency. Our approach was evaluated on small-sized public benchmarks and a newly developed large-scale benchmark inspired by the Taillard benchmark. The results show that our approach matches traditional methods on smaller instances and outperforms them on larger ones in terms of makespan while achieving a tenfold reduction in computation time. To ensure reproducibility, we propose a gym-compatible environment and an instance generator. Additionally, an ablation study evaluates the contribution of each framework component to its overall performance.

Flexible Manufacturing Systems Intralogistics: Dynamic Optimization of AGVs and Tool Sharing Using Coloured-Timed Petri Nets and Actor-Critic RL with Actions Masking

TL;DR

This work tackles the complex scheduling of flexible manufacturing systems (FMS) with constrained material handling via automated guided vehicles (AGVs) and tool sharing. It proposes a hybrid framework that combines Colored-Timed Petri Nets (CTPNs) with an actor-critic reinforcement learning approach using action masking (Maskable PPO) to manage the expanded action space. Key contributions include a modular PetriNet-based model, a gym-compatible large-scale Taillard-inspired benchmark, and a thorough ablation study confirming the roles of reward shaping, lookahead, and masking. Results show that the proposed PetriRL method matches traditional methods on small instances and substantially outperforms them on large-scale problems, with dramatic reductions in computation time and strong generalization across varied instance sizes, enabling dynamic adaptation to late orders and disturbances.

Abstract

Flexible Manufacturing Systems (FMS) are pivotal in optimizing production processes in today's rapidly evolving manufacturing landscape. This paper advances the traditional job shop scheduling problem by incorporating additional complexities through the simultaneous integration of automated guided vehicles (AGVs) and tool-sharing systems. We propose a novel approach that combines Colored-Timed Petri Nets (CTPNs) with actor-critic model-based reinforcement learning (MBRL), effectively addressing the multifaceted challenges associated with FMS. CTPNs provide a formal modeling structure and dynamic action masking, significantly reducing the action search space, while MBRL ensures adaptability to changing environments through the learned policy. Leveraging the advantages of MBRL, we incorporate a lookahead strategy for optimal positioning of AGVs, improving operational efficiency. Our approach was evaluated on small-sized public benchmarks and a newly developed large-scale benchmark inspired by the Taillard benchmark. The results show that our approach matches traditional methods on smaller instances and outperforms them on larger ones in terms of makespan while achieving a tenfold reduction in computation time. To ensure reproducibility, we propose a gym-compatible environment and an instance generator. Additionally, an ablation study evaluates the contribution of each framework component to its overall performance.
Paper Structure (26 sections, 4 equations, 13 figures, 5 tables, 2 algorithms)

This paper contains 26 sections, 4 equations, 13 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Background and Preliminaries
  4. Reinforcement Learning
  5. Petri Nets
  6. Problem and Modelling
  7. Problem Definitions
  8. Key Assumptions
  9. Constraints
  10. Integrating PetriRL for intralogistics
  11. FMS Environment using Colored-Timed Petri Nets
  12. RL environment and Maskable PPO
  13. AGVs transport time calculation and lookaheads
  14. Results and Discussion
  15. Large instance benchmark
  16. ...and 11 more sections

Figures (13)

  • Figure 1: Examples of Petri Net Variants: (a) basic Petri Net, (b) colored Petri Net, and (c) colored-timed Petri Net.
  • Figure 2: An FSM problem example Reddy.2021
  • Figure 3: Colored-Timed Petri Nets modelling a FSM problems
  • Figure 4: Evaluation of Replicated Algorithm Performance Against the Original Implementation. Results from three runs of the replicated algorithm are compared to the original results to account for randomness.
  • Figure 5: Training and deployment time for different instances. The red line represents the agent training time requirement for different instance sizes, as shown on the corresponding y-axis on the right. The blue line represents the time required to solve the FSM problem during the inference. The corresponding y-axis is on the left. The dotted grey line represents the increasing number of training steps in every size group.
  • ...and 8 more figures