Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In the Wild Ungraspable Object Picking with Bimanual Nonprehensile Manipulation

Albert Wu, Dan Kruse

TL;DR

This paper focuses on a grocery shopping scenario, where a bimanual mobile manipulator equipped with a suction gripper and a PJG is tasked with re-trieving ungraspable items from tightly packed grocery shelves and highlights the potential of bimanual nonprehensile manipulation for in-the-wild robotic picking tasks.

Abstract

Picking diverse objects in the real world is a fundamental robotics skill. However, many objects in such settings are bulky, heavy, or irregularly shaped, making them ungraspable by conventional end effectors like suction grippers and parallel jaw grippers (PJGs). In this paper, we expand the range of pickable items without hardware modifications using bimanual nonprehensile manipulation. We focus on a grocery shopping scenario, where a bimanual mobile manipulator equipped with a suction gripper and a PJG is tasked with retrieving ungraspable items from tightly packed grocery shelves. From visual observations, our method first identifies optimal grasp points based on force closure and friction constraints. If the grasp points are occluded, a series of nonprehensile nudging motions are performed to clear the obstruction. A bimanual grasp utilizing contacts on the side of the end effectors is then executed to grasp the target item. In our replica grocery store, we achieved a 90% success rate over 102 trials in uncluttered scenes, and a 67% success rate over 45 trials in cluttered scenes. We also deployed our system to a real-world grocery store and successfully picked previously unseen items. Our results highlight the potential of bimanual nonprehensile manipulation for in-the-wild robotic picking tasks. A video summarizing this work can be found at youtu.be/g0hOrDuK8jM

In the Wild Ungraspable Object Picking with Bimanual Nonprehensile Manipulation

TL;DR

This paper focuses on a grocery shopping scenario, where a bimanual mobile manipulator equipped with a suction gripper and a PJG is tasked with re-trieving ungraspable items from tightly packed grocery shelves and highlights the potential of bimanual nonprehensile manipulation for in-the-wild robotic picking tasks.

Abstract

Picking diverse objects in the real world is a fundamental robotics skill. However, many objects in such settings are bulky, heavy, or irregularly shaped, making them ungraspable by conventional end effectors like suction grippers and parallel jaw grippers (PJGs). In this paper, we expand the range of pickable items without hardware modifications using bimanual nonprehensile manipulation. We focus on a grocery shopping scenario, where a bimanual mobile manipulator equipped with a suction gripper and a PJG is tasked with retrieving ungraspable items from tightly packed grocery shelves. From visual observations, our method first identifies optimal grasp points based on force closure and friction constraints. If the grasp points are occluded, a series of nonprehensile nudging motions are performed to clear the obstruction. A bimanual grasp utilizing contacts on the side of the end effectors is then executed to grasp the target item. In our replica grocery store, we achieved a 90% success rate over 102 trials in uncluttered scenes, and a 67% success rate over 45 trials in cluttered scenes. We also deployed our system to a real-world grocery store and successfully picked previously unseen items. Our results highlight the potential of bimanual nonprehensile manipulation for in-the-wild robotic picking tasks. A video summarizing this work can be found at youtu.be/g0hOrDuK8jM
Paper Structure (20 sections, 4 theorems, 5 equations, 6 figures, 2 tables, 2 algorithms)

This paper contains 20 sections, 4 theorems, 5 equations, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Robot system
  4. "Shopping for ungraspable grocery" task
  5. Nonprehensile bimanual picking
  6. Perception
  7. Planning bimanual grasp points
  8. Formulating the grasp quality metric $l_g$
  9. Computing $l_g$ in practice
  10. Planning declutter motion
  11. Ranking declutter-grasp plans
  12. Hardware execution
  13. Evaluation
  14. Test scene selection
  15. Evaluation protocol
  16. ...and 5 more sections

Key Result

Lemma 1

For a given $\bm{w}\in\bm{W}$, $\exists \bm{w}_1, \bm{w}_2 \in \partial\bm{W}$ such that $w_{\tau}=w_{\tau 1}= w_{\tau 2}, \bm{w} = \beta\bm{w}_1 + (1-\beta)\bm{w}_2$ for some $\beta \in [0,1]$. Moreoever, $l_g(\cdot; \bm{w}) \leq \max\left\{l_g(\cdot; \bm{w}_1), l_g(\cdot; \bm{w}_2)\right\}.$

Figures (6)

  • Figure 1: TTT picking items from clutter at our mock grocery replica store (\ref{['subfig:bottom_shelf_pick']}-\ref{['subfig:top_shelf_pick']}) and a real-world grocery store(\ref{['subfig:real_store_pick']}).
  • Figure 2: \ref{['fig:frame_definitions']}: Frame and shelf dimension definitions. \ref{['subfig:mock_grocery']}, \ref{['subfig:real_grocery']}: Test environments. Red arrows in \ref{['subfig:real_grocery']}: target items.
  • Figure 3: Nonprehensile picking pipeline. \ref{['subfig:item_detection']} shows the item detection in the head camera image (brown box) and the segmented target item point cloud (cyan). \ref{['subfig:grasp_planning']} shows the shelf (yellow), fitted alpha shape (orange), and grasps candidates $\overline{\bm{c}_l\bm{c}_r}$ (red: rejected, cyan: in $\mathcal{C}$). \ref{['subfig:nudge_planning']} shows a declutter planning scenario. $t$: target item. $n$: item adjacent to the target item. $h$: adjacent item at end effector height that may cause occlusion. $e$: end effector. $l,r$: left and right side. $\bm{S}$: shelf voxels. The $(\bm{c}_l,\bm{c}_r)$ choice in magenta is not occluded. \ref{['subfig:grasp_execution']} shows unobstructed grasp approach to the $(\bm{c}_l,\bm{c}_r)$ shown in \ref{['subfig:nudge_planning']}.
  • Figure 4: Nudging sequence. \ref{['subfig:nudge_insert']} shows the second insertion in the 4-nudge sequence, where the gap between items remains minimal. \ref{['subfig:nudge_push']} and \ref{['subfig:nudge_retract']} illustrate the third push and retract actions. At this point, the gap becomes clearly visible. Following this sequence, sufficient clearance is created for the gripper to approach the grasp points, as shown in \ref{['subfig:grasp_insert']}.
  • Figure 5: Item physical properties and freespace picking trial success rate. Red: 1/3. Orange: 2/3. Green: 3/3. Narrower and heavier items failed more frequently.
  • ...and 1 more figures

Theorems & Definitions (8)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 1
  • proof
  • Corollary 1.1
  • proof