Multi-Objective Reinforcement Learning for Large-Scale Tote Allocation in Human-Robot Collaborative Fulfillment Centers

TL;DR

This work builds on recent theoretical advances in solving constrained RL problems via best-response and no-regret dynamics in zero-sum games, enabling principled minimax policy learning and demonstrates the promise of MORL in solving complex, high-impact decision-making problems in large-scale industrial systems.

Abstract

Optimizing the consolidation process in container-based fulfillment centers requires trading off competing objectives such as processing speed, resource usage, and space utilization while adhering to a range of real-world operational constraints. This process involves moving items between containers via a combination of human and robotic workstations to free up space for inbound inventory and increase container utilization. We formulate this problem as a large-scale Multi-Objective Reinforcement Learning (MORL) task with high-dimensional state spaces and dynamic system behavior. Our method builds on recent theoretical advances in solving constrained RL problems via best-response and no-regret dynamics in zero-sum games, enabling principled minimax policy learning. Policy evaluation on realistic warehouse simulations shows that our approach effectively trades off objectives, and we empirically observe that it learns a single policy that simultaneously satisfies all constraints, even if this is not theoretically guaranteed. We further introduce a theoretical framework to handle the problem of error cancellation, where time-averaged solutions display oscillatory behavior. This method returns a single iterate whose Lagrangian value is close to the minimax value of the game. These results demonstrate the promise of MORL in solving complex, high-impact decision-making problems in large-scale industrial systems.

Figures (8)

• Figure 1: Left: High-throughput container-based fulfillment center with dense human-robot collaboration. Right: Robotic consolidation station autonomously transferring items between totes.
• Figure 2: Human-robot collaborative fulfillment center workflow.
• Figure 3: Episodic returns for DQN in the single-objective ETPH setting, showing unnormalized performance of a best-response policy optimizing ETPH.
• Figure 4: Average Lagrange multipliers $\bar{\lambda}$ over training rounds. The $N_{\texttt{large}}$ and robot capacity multipliers remain at zero, while the remaining constraints exhibit significant oscillations.
• Figure 5: Policy performance over repeated game rounds. (a) Performance of individual policies at each round. (b) Performance of time averaged policies. Each plot shows the global ETPH objective and four constraint values. Black horizontal lines denote constraint thresholds, red markers indicate violations, and green vertical regions indicate rounds where all constraints are simultaneously satisfied.

Theorems & Definitions (9)

• Definition 1: $\nu$-approximate Minimax Equilibrium
• Theorem 3.1: (Informal) Best-Response vs. No-Regret Freund1996GameTO
• Definition 2: $\epsilon$-approximate best-response
• Theorem 3.2: Online Gradient Descent No-Regret zinkevich
• Definition 3: Episodic Occupancy Measure
• Lemma 1.1: Uniform Concentration for $\widehat{L}$
• proof
• Theorem 1.1: Single Iterate Extraction
• proof