Solving Sequential Manipulation Puzzles by Finding Easier Subproblems

TL;DR

This work addresses sequential manipulation in cluttered, constrained environments by coupling a forward search over auxiliary pick-and-place subproblems with an optimization-based Task-and-Motion Planner (Logic-Geometric Programming). Subgoals are generated and prioritized with multiple strategies and solved via a nonlinear optimizer plus sampling-based planning, feeding feasible configurations back to the main LGP solver. Experiments on both manually designed and procedurally generated scenes demonstrate substantial gains over a baseline MBTS solver, particularly when the goal is occluded, highlighting robustness for complex sequential manipulation. The approach generalizes beyond pick-and-place and offers a practical path toward reliable manipulation in narrow, obstacle-filled spaces.

Abstract

We consider a set of challenging sequential manipulation puzzles, where an agent has to interact with multiple movable objects and navigate narrow passages. Such settings are notoriously difficult for Task-and-Motion Planners, as they require interdependent regrasps and solving hard motion planning problems. In this paper, we propose to search over sequences of easier pick-and-place subproblems, which can lead to the solution of the manipulation puzzle. Our method combines a heuristic-driven forward search of subproblems with an optimization-based Task-and-Motion Planning solver. To guide the search, we introduce heuristics to generate and prioritize useful subgoals. We evaluate our approach on various manually designed and automatically generated scenes, demonstrating the benefits of auxiliary subproblems in sequential manipulation planning.

Figures (6)

• Figure 1: Solving the Wall pick-and-place puzzle with subgoals. In order to bring the goal-object (blue) to the goal (red square), the agent (yellow circle) moves it and regrasps at several intermediate subgoal locations (gray dots).
• Figure 2: Subproblem sequence. Solved subproblems and problems in green, unsolved in red.
• Figure 3: Subgoals in the scene Wall. a) Rnd b) Rnd filtered by heuristics. c) Btl d) Btl after filtering e) Hum
• Figure 4: Puzzle scenes: the agent (yellow circle) needs to bring the goal-object (blue) to the goal (red square). Movable obstacles in white, static walls in brown.
• Figure 5: Proceduraly Generated Scenes (subset) with walls (brown) and two movable objects: the goal-object (blue) and an obstacle-object (pink).