MARL Warehouse Robots

TL;DR

The paper evaluates multi-agent reinforcement learning for cooperative warehouse robotics, comparing QMIX, IPPO, and MASAC across the MPE benchmark, RWARE, and Unity 3D deployment. It finds QMIX's value-decomposition approach yields strong performance on sparse-reward coordination but demands extended hyperparameter tuning and large training budgets. Scaling to larger robot teams remains a major challenge, potentially requiring hierarchical architectures. A Unity ML-Agents integration demonstrates sim-to-sim transfer, and the authors provide code and data for reproducibility.

Abstract

We present a comparative study of multi-agent reinforcement learning (MARL) algorithms for cooperative warehouse robotics. We evaluate QMIX and IPPO on the Robotic Warehouse (RWARE) environment and a custom Unity 3D simulation. Our experiments reveal that QMIX's value decomposition significantly outperforms independent learning approaches (achieving 3.25 mean return vs. 0.38 for advanced IPPO), but requires extensive hyperparameter tuning -- particularly extended epsilon annealing (5M+ steps) for sparse reward discovery. We demonstrate successful deployment in Unity ML-Agents, achieving consistent package delivery after 1M training steps. While MARL shows promise for small-scale deployments (2-4 robots), significant scaling challenges remain. Code and analyses: https://pallman14.github.io/MARL-QMIX-Warehouse-Robots/