Demonstrating Mobile Manipulation in the Wild: A Metrics-Driven Approach

TL;DR

The paper presents a general-purpose mobile manipulation system (TTT) validated through six long-term field tests in a real unmodified grocery store, using a data-driven, end-to-end evaluation framework. It combines a learned stereo depth pipeline, voxel-based mapping, advanced object perception, and a modular planning stack (task, navigation, motion, and grasp planning) to achieve autonomous shopping with two end-effectors. Key contributions include a comprehensive perception-localization pipeline, a DRM-based motion planner for a highly redundant robot, and a rigorous, quarterly field-test methodology with concrete reliability, shopping performance, and speed metrics. The study demonstrates the practical challenges of deploying mobile manipulators in the real world, underscores the importance of end-to-end testing, and provides actionable insights for improving robustness and transitioning research methods toward real-world impact.

Abstract

We present our general-purpose mobile manipulation system consisting of a custom robot platform and key algorithms spanning perception and planning. To extensively test the system in the wild and benchmark its performance, we choose a grocery shopping scenario in an actual, unmodified grocery store. We derive key performance metrics from detailed robot log data collected during six week-long field tests, spread across 18 months. These objective metrics, gained from complex yet repeatable tests, drive the direction of our research efforts and let us continuously improve our system's performance. We find that thorough end-to-end system-level testing of a complex mobile manipulation system can serve as a reality-check for state-of-the-art methods in robotics. This effectively grounds robotics research efforts in real world needs and challenges, which we deem highly useful for the advancement of the field. To this end, we share our key insights and takeaways to inspire and accelerate similar system-level research projects.

Figures (10)

• Figure 1: TTT operating in a real, unmodified grocery store.
• Figure 2: Overview of our interconnected, modular software system designed for mobile manipulation tasks.
• Figure 3: The object perception pipeline as outlined in Section \ref{['sec:object']}.
• Figure 4: (a) Output of a state-of-the-art BA algorithm Agarwal-2022. Notice the noise and sparsity of the reconstruction. (b) Our map generated by registering the stereo-based 3D pointclouds using the poses computed by the same BA algorithm. The green rectangles show a side-to-side comparison between the same two regions in the maps and highlight the difference in reconstruction quality. The positions of the mapped store items are rendered as blue spheres and their orientations are indicated by red, blue and green coordinate frames.
• Figure 5: Highly simplified task summary. Note that our hierarchical finite state machine includes many more states/transitions, but this provides a pictorial description of the main components of the grocery task.