Practical Insights on Grasp Strategies for Mobile Manipulation in the Wild

TL;DR

This work addresses the challenge of reliable mobile manipulation in unstructured real-world environments by presenting SHOPPER, a fully on-robot platform deployed in an actual grocery store to study practical grasping strategies. It details a complete grasping pipeline and a thorough failure analysis from large-scale field tests, highlighting the diverse strategies needed to pick items from shelves, bags, and produce piles. The paper identifies eight fundamental failure modes and discusses actionable improvements, emphasizing real-time tracking, dual-arm coordination, and robust perception and planning under wild conditions. Practically, this work provides a robust platform for ongoing development and a data-rich corpus to guide future learning-driven approaches in real-world mobile manipulation.

Abstract

Mobile manipulation robots are continuously advancing, with their grasping capabilities rapidly progressing. However, there are still significant gaps preventing state-of-the-art mobile manipulators from widespread real-world deployments, including their ability to reliably grasp items in unstructured environments. To help bridge this gap, we developed SHOPPER, a mobile manipulation robot platform designed to push the boundaries of reliable and generalizable grasp strategies. We develop these grasp strategies and deploy them in a real-world grocery store -- an exceptionally challenging setting chosen for its vast diversity of manipulable items, fixtures, and layouts. In this work, we present our detailed approach to designing general grasp strategies towards picking any item in a real grocery store. Additionally, we provide an in-depth analysis of our latest real-world field test, discussing key findings related to fundamental failure modes over hundreds of distinct pick attempts. Through our detailed analysis, we aim to offer valuable practical insights and identify key grasping challenges, which can guide the robotics community towards pressing open problems in the field.

Figures (7)

• Figure 1: SHOPPER is a general-purpose mobile manipulator capable of executing complex grasp strategies, such as retrieving an item hanging on a hook behind a fridge door.
• Figure 2: Overview of our modular, interconnected software system for mobile manipulation.
• Figure 3: SHOPPER retrieves items from a wide variety of different fixtures and item arrangements.
• Figure 4: Overview of the SHOPPER grasp module.
• Figure 5: Examples of the diverse range of grasp strategies with robot perception visualized at top: (a) a suction grasp that optimizes the most stable attachment point; (b) a gripper grasp that locates secure contact points using the 2D outline of the shelved item; (c) a gripper grasp planned with a shape-completed hull of the target item (see inset) for picking from produce piles; (d) a handled-item grasp that identifies keypoints to parameterize gripper insertion into the handle; (e) a side grasp designed for bags, where neighboring items are first swiped away to provide the gripper access to the bag's side.