Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficiently Manipulating Clutter via Learning and Search-Based Reasoning

Baichuan Huang

TL;DR

This work tackles robust, long-horizon robotic object rearrangement in clutter by fusing learning-based interaction prediction with search-based planning and GPU-accelerated parallelization. Central components include DIPN for push-prediction, GN for grasp evaluation, and multi-branch planning frameworks (VFT, MORE, PMBS) that enable efficient, scalable planning across cluttered scenes. Key contributions reveal: (i) DIPN achieving >90% pose-forecast accuracy; (ii) VFT delivering high-coverage, multi-step planning with DIPN+GN; (iii) PMBS delivering >30× planning speedups with near real-time performance; and (iv) HBFS and PMMR enabling effective multi-primitive tabletop rearrangement. The results demonstrate strong sim-to-real transfer, state-of-the-art performance on cluttered retrieval and rearrangement tasks, and practical implications for autonomous warehouse and household robotics, with future work aiming at broader action sets and further acceleration.

Abstract

This thesis presents novel algorithms to advance robotic object rearrangement, a critical task for autonomous systems in applications like warehouse automation and household assistance. Addressing challenges of high-dimensional planning, complex object interactions, and computational demands, our work integrates deep learning for interaction prediction, tree search for action sequencing, and parallelized computation for efficiency. Key contributions include the Deep Interaction Prediction Network (DIPN) for accurate push motion forecasting (over 90% accuracy), its synergistic integration with Monte Carlo Tree Search (MCTS) for effective non-prehensile object retrieval (100% completion in specific challenging scenarios), and the Parallel MCTS with Batched Simulations (PMBS) framework, which achieves substantial planning speed-up while maintaining or improving solution quality. The research further explores combining diverse manipulation primitives, validated extensively through simulated and real-world experiments.

Efficiently Manipulating Clutter via Learning and Search-Based Reasoning

TL;DR

This work tackles robust, long-horizon robotic object rearrangement in clutter by fusing learning-based interaction prediction with search-based planning and GPU-accelerated parallelization. Central components include DIPN for push-prediction, GN for grasp evaluation, and multi-branch planning frameworks (VFT, MORE, PMBS) that enable efficient, scalable planning across cluttered scenes. Key contributions reveal: (i) DIPN achieving >90% pose-forecast accuracy; (ii) VFT delivering high-coverage, multi-step planning with DIPN+GN; (iii) PMBS delivering >30× planning speedups with near real-time performance; and (iv) HBFS and PMMR enabling effective multi-primitive tabletop rearrangement. The results demonstrate strong sim-to-real transfer, state-of-the-art performance on cluttered retrieval and rearrangement tasks, and practical implications for autonomous warehouse and household robotics, with future work aiming at broader action sets and further acceleration.

Abstract

This thesis presents novel algorithms to advance robotic object rearrangement, a critical task for autonomous systems in applications like warehouse automation and household assistance. Addressing challenges of high-dimensional planning, complex object interactions, and computational demands, our work integrates deep learning for interaction prediction, tree search for action sequencing, and parallelized computation for efficiency. Key contributions include the Deep Interaction Prediction Network (DIPN) for accurate push motion forecasting (over 90% accuracy), its synergistic integration with Monte Carlo Tree Search (MCTS) for effective non-prehensile object retrieval (100% completion in specific challenging scenarios), and the Parallel MCTS with Batched Simulations (PMBS) framework, which achieves substantial planning speed-up while maintaining or improving solution quality. The research further explores combining diverse manipulation primitives, validated extensively through simulated and real-world experiments.
Paper Structure (95 sections, 14 equations, 47 figures, 14 tables, 6 algorithms)

This paper contains 95 sections, 14 equations, 47 figures, 14 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. DIPN: Deep Interaction Prediction Network with Application to Clutter Removal
  3. Visual Foresight Trees for Object Retrieval from Clutter with Nonprehensile Rearrangement
  4. Interleaving Monte Carlo Tree Search and Self-Supervised Learning for Object Retrieval in Clutter
  5. Parallel Monte Carlo Tree Search with Batched Rigid-body Simulations for Speeding up Long-Horizon Episodic Robot Planning
  6. Toward Optimal Tabletop Rearrangement with Multiple Manipulation Primitives
  7. Related Works
  8. Prehensile vs. Non-Prehensile Manipulation
  9. Object Grasping
  10. Analytical vs. Data-Driven Approaches
  11. Grasping in Dense Environments
  12. Pushing
  13. Push-Grasping
  14. Singulation
  15. Definition and Techniques
  16. ...and 80 more sections

Figures (47)

  • Figure 1: Examples of robot manipulation tasks. (a) Grasping objects from clutter: This can involve grasping all objects or targeting a specific item. The challenge lies in creating sufficient space for the gripper to access objects. (b) Dynamic grasping: Manipulating moving objects, such as receiving an item from a human hand. (c) Object rearrangement: Reorganizing objects to achieve a desired layout, similar to housekeeping. This task requires both high-level planning and precise motion control.
  • Figure 2: Structure of the dissertation. \ref{['chap:dipn']} introduces the Deep Interaction Prediction Network for one-step push prediction in clutter removal. \ref{['chap:vft']} extends to multi-step planning for efficient object retrieval using push actions. \ref{['chap:more', 'chap:pmbs']} explore GPU-accelerated Monte Carlo tree search: \ref{['chap:more']} focuses on learning a strategic network to guide tree search, while \ref{['chap:pmbs']} utilizes Isaac Gym for parallel simulations in real robot execution. \ref{['chap:remp']} applies similar concepts to object rearrangement in constrained spaces, incorporating motion planning.
  • Figure 3: (a) The system setup includes a workspace with objects to remove, a Universal Robots UR-5e manipulator with a Robotiq 2F-85 two-finger gripper, and an Intel RealSense D435 RGB-D camera. (b) An example push action and superimposed images of scenes before and after the push. (c) System architecture of our pipeline, and one predicted image that DIPN can generate for the push shown in (b). Notice the similarity between the predicted synthetic image and the real image resulting from the push action.
  • Figure 4: DIPN flow with an example. The network components dedicated to an object are color-coded to match the object. We only show the full network for the blue triangle object; the instance-specific structures for the other objects share the same weights and are simplified as dashed lines. Components inside the orange dotted line are the core of the DIPN. The output image is synthesized by applying the predicted transformations to the object segments.
  • Figure 5: Sampled action in purple arrows around each object.
  • ...and 42 more figures