Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping

Guangyi Liu, Suzan Iloglu, Michael Caldara, Joseph W. Durham, Michael M. Zavlanos

TL;DR

This work addresses robust destination-to-chute mapping in Amazon-style sortation under dynamic induction patterns. It introduces DRMARL, which combines group distributionally robust optimization with MARL and a contextual-bandit-based worst-case predictor to guard against distribution shifts in induction. The approach yields near-optimal, distribution-robust chute mappings while substantially reducing training cost and maintaining scalability, demonstrated by an average recirculation reduction of around 80% and notable throughput gains. The methodology generalizes beyond sortation to other large-scale, distribution-shift-prone MARL applications, offering a practical pathway to robust, efficient multi-agent control in industrial settings.

Abstract

In Amazon robotic warehouses, the destination-to-chute mapping problem is crucial for efficient package sorting. Often, however, this problem is complicated by uncertain and dynamic package induction rates, which can lead to increased package recirculation. To tackle this challenge, we introduce a Distributionally Robust Multi-Agent Reinforcement Learning (DRMARL) framework that learns a destination-to-chute mapping policy that is resilient to adversarial variations in induction rates. Specifically, DRMARL relies on group distributionally robust optimization (DRO) to learn a policy that performs well not only on average but also on each individual subpopulation of induction rates within the group that capture, for example, different seasonality or operation modes of the system. This approach is then combined with a novel contextual bandit-based predictor of the worst-case induction distribution for each state-action pair, significantly reducing the cost of exploration and thereby increasing the learning efficiency and scalability of our framework. Extensive simulations demonstrate that DRMARL achieves robust chute mapping in the presence of varying induction distributions, reducing package recirculation by an average of 80\% in the simulation scenario.

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping

TL;DR

This work addresses robust destination-to-chute mapping in Amazon-style sortation under dynamic induction patterns. It introduces DRMARL, which combines group distributionally robust optimization with MARL and a contextual-bandit-based worst-case predictor to guard against distribution shifts in induction. The approach yields near-optimal, distribution-robust chute mappings while substantially reducing training cost and maintaining scalability, demonstrated by an average recirculation reduction of around 80% and notable throughput gains. The methodology generalizes beyond sortation to other large-scale, distribution-shift-prone MARL applications, offering a practical pathway to robust, efficient multi-agent control in industrial settings.

Abstract

In Amazon robotic warehouses, the destination-to-chute mapping problem is crucial for efficient package sorting. Often, however, this problem is complicated by uncertain and dynamic package induction rates, which can lead to increased package recirculation. To tackle this challenge, we introduce a Distributionally Robust Multi-Agent Reinforcement Learning (DRMARL) framework that learns a destination-to-chute mapping policy that is resilient to adversarial variations in induction rates. Specifically, DRMARL relies on group distributionally robust optimization (DRO) to learn a policy that performs well not only on average but also on each individual subpopulation of induction rates within the group that capture, for example, different seasonality or operation modes of the system. This approach is then combined with a novel contextual bandit-based predictor of the worst-case induction distribution for each state-action pair, significantly reducing the cost of exploration and thereby increasing the learning efficiency and scalability of our framework. Extensive simulations demonstrate that DRMARL achieves robust chute mapping in the presence of varying induction distributions, reducing package recirculation by an average of 80\% in the simulation scenario.

Paper Structure

This paper contains 28 sections, 2 theorems, 37 equations, 11 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Problem Formulation
  4. MARL Structure
  5. Induction Distribution
  6. Dimension Reduction of the State-Action Space
  7. Feasibility of Joint Actions
  8. Distributionally Robust Multi-Agent Reinforcement Learning with Group DRO
  9. Group DRO
  10. DRMARL with Group DRO
  11. Contextual Bandit-based Worst-Case Reward Predictor for DRMARL
  12. CB-based Worst-Case Reward Predictor
  13. DRMARL with CB-based Worst-Case Reward Predictor
  14. Simulation
  15. Simplified Robotic Sortation Warehouse Simulation
  16. ...and 13 more sections

Key Result

Lemma 3.1

Consider an ambiguity set $\mathfrak{M}$ formed by $\mathbb{P}_g$s as defined in eq:group_ambi. For any state-action pair $(s,a) \in \mathcal{S}\times \mathcal{A}$, the worst-case expected reward satisfies: where $\mathcal{G}$ denotes the set of group indices.

Figures (11)

  • Figure 1: Schematic layout of an Amazon robotic sortation warehouse featuring eject chutes.
  • Figure 2: Flow of packages in the Amazon robotic sortation warehouse.
  • Figure 3: Ambiguity set $\mathfrak M$ in regular DRO rahimian2019distributionally (left) versus group DRO sagawa2019distributionally (right).
  • Figure 4: Training efficiency comparison in simplified warehouse.
  • Figure 5: Training recirculation rate prediction loss \ref{['eq:cb_loss']} for CB in simplified robotic sortation warehouse.
  • ...and 6 more figures

Theorems & Definitions (3)

  • Definition 2.1
  • Lemma 3.1
  • Lemma 3.2