Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning for AMR Charging Decisions: The Impact of Reward and Action Space Design

Janik Bischoff, Alexandru Rinciog, Anne Meyer

TL;DR

This work tackles the problem of optimizing AMR charging in large-scale block-stacking warehouses using reinforcement learning. It extends the SLAPStack framework with battery management, formulates a Markov decision process for joint charging decisions and durations, and evaluates multiple reward and action-space configurations using PPO on the WEPAStacks benchmark. The results reveal a trade-off: flexible, unconstrained RL designs can uncover high-performing strategies but may converge slowly and be unstable, whereas domain-guided reward shaping and reduced action spaces improve learning stability but may limit exploration and generalization. Overall, RL-based charging strategies can outperform strong heuristics on realistic, large-scale instances, with the choice of reward design and action masking playing a pivotal role in practical performance and reproducibility.

Abstract

We propose a novel reinforcement learning (RL) design to optimize the charging strategy for autonomous mobile robots in large-scale block stacking warehouses. RL design involves a wide array of choices that can mostly only be evaluated through lengthy experimentation. Our study focuses on how different reward and action space configurations, ranging from flexible setups to more guided, domain-informed design configurations, affect the agent performance. Using heuristic charging strategies as a baseline, we demonstrate the superiority of flexible, RL-based approaches in terms of service times. Furthermore, our findings highlight a trade-off: While more open-ended designs are able to discover well-performing strategies on their own, they may require longer convergence times and are less stable, whereas guided configurations lead to a more stable learning process but display a more limited generalization potential. Our contributions are threefold. First, we extend SLAPStack, an open-source, RL-compatible simulation-framework to accommodate charging strategies. Second, we introduce a novel RL design for tackling the charging strategy problem. Finally, we introduce several novel adaptive baseline heuristics and reproducibly evaluate the design using a Proximal Policy Optimization agent and varying different design configurations, with a focus on reward.

Reinforcement Learning for AMR Charging Decisions: The Impact of Reward and Action Space Design

TL;DR

This work tackles the problem of optimizing AMR charging in large-scale block-stacking warehouses using reinforcement learning. It extends the SLAPStack framework with battery management, formulates a Markov decision process for joint charging decisions and durations, and evaluates multiple reward and action-space configurations using PPO on the WEPAStacks benchmark. The results reveal a trade-off: flexible, unconstrained RL designs can uncover high-performing strategies but may converge slowly and be unstable, whereas domain-guided reward shaping and reduced action spaces improve learning stability but may limit exploration and generalization. Overall, RL-based charging strategies can outperform strong heuristics on realistic, large-scale instances, with the choice of reward design and action masking playing a pivotal role in practical performance and reproducibility.

Abstract

We propose a novel reinforcement learning (RL) design to optimize the charging strategy for autonomous mobile robots in large-scale block stacking warehouses. RL design involves a wide array of choices that can mostly only be evaluated through lengthy experimentation. Our study focuses on how different reward and action space configurations, ranging from flexible setups to more guided, domain-informed design configurations, affect the agent performance. Using heuristic charging strategies as a baseline, we demonstrate the superiority of flexible, RL-based approaches in terms of service times. Furthermore, our findings highlight a trade-off: While more open-ended designs are able to discover well-performing strategies on their own, they may require longer convergence times and are less stable, whereas guided configurations lead to a more stable learning process but display a more limited generalization potential. Our contributions are threefold. First, we extend SLAPStack, an open-source, RL-compatible simulation-framework to accommodate charging strategies. Second, we introduce a novel RL design for tackling the charging strategy problem. Finally, we introduce several novel adaptive baseline heuristics and reproducibly evaluate the design using a Proximal Policy Optimization agent and varying different design configurations, with a focus on reward.
Paper Structure (21 sections, 4 figures, 4 tables)

This paper contains 21 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning Free Approaches
  4. Reinforcement Learning Approaches
  5. Benchmark Extensions and Baseline Evaluation
  6. Benchmark Framework Extension
  7. Augmented SLAPStack
  8. Augmented WEPAStacks
  9. Baseline Strategies
  10. Definition
  11. Evaluation
  12. A Reinforcement Learning Charging Strategy
  13. State Representation and Feature Design
  14. Reward and Action Space Design
  15. Model
  16. ...and 6 more sections

Figures (4)

  • Figure 1: The problems associated with battery management as per Wang2014.
  • Figure 2: WEPAStacks use-case data visualization.
  • Figure 3: Week and intercept dependent distribution of service times and buffer sizes.
  • Figure 4: Learning metric evolution during training for the proposed configurations.