Iterative Shaping of Multi-Particle Aggregates based on Action Trees and VLM

TL;DR

The paper tackles the challenge of guiding dispersed particles into a target region while preserving group cohesion using a bimanual robot. It introduces a Fourier descriptor-based contour representation, a cohesiveness metric, and an Iterative Action Tree for macro-scale path planning, integrated with an MPC-based micro-scale trajectory refinement and a vision-language grounded high-level planner. The approach is validated on real robots, showing the system can shape and shepherd multi-particle aggregates and maintain cohesion, achieving performance close to human levels. This work advances cohesive multi-object manipulation by combining contour-based shape control, shepherding-style planning, and vision-language grounding for robust task execution in cluttered environments.

Abstract

In this paper, we address the problem of manipulating multi-particle aggregates using a bimanual robotic system. Our approach enables the autonomous transport of dispersed particles through a series of shaping and pushing actions using robotically-controlled tools. Achieving this advanced manipulation capability presents two key challenges: high-level task planning and trajectory execution. For task planning, we leverage Vision Language Models (VLMs) to enable primitive actions such as tool affordance grasping and non-prehensile particle pushing. For trajectory execution, we represent the evolving particle aggregate's contour using truncated Fourier series, providing efficient parametrization of its closed shape. We adaptively compute trajectory waypoints based on group cohesion and the geometric centroid of the aggregate, accounting for its spatial distribution and collective motion. Through real-world experiments, we demonstrate the effectiveness of our methodology in actively shaping and manipulating multi-particle aggregates while maintaining high system cohesion.

Figures (8)

• Figure 1: Setup of the addressed multi-particle shaping task.
• Figure 2: System Structure: The system consists of a Task Planner with a Shape Manipulation and Control module. An ontological knowledge graph and VLM describe the scene, interpreted by an LLM for task decomposition. Shape is represented by Fourier Series to generate an action tree based on particle cohesiveness. An MPC guides manipulation while avoiding obstacles, with the VLM confirming particle status if detection becomes difficult.
• Figure 3: Conceptual representation of the proposed iterative action tree. (1)--(3): The orange point represents a waypoint on a trajectory, which is positioned at the centroid of a triangle. (4): The overall trajectory can be obtained by applying graph theory principles to connect these strategically placed waypoints.
• Figure 4: Before the micro-scale shape refinement, the tooltip (the purple dashed trajectory) will hit the grey wall. After the refinement, the tooltip (the red dashed trajectory) will be smoothly avoiding the grey wall.
• Figure 5: Experiments: (a) Different shapes and amounts of particles are used to evaluate the performance of the proposed shape representation method. The contour of the particle is expanded and shown in blue lines. The black points are the waypoints of the yellow trajectories. The brown circle represents the gate point which is towards the grey box; (b) The evolution of the shape described by Fourier-based Representation; (c) The evolutionary changes in distance between the centroid of the particle group with the gate and the changes of the group size; (d) The evolution of the number of particles remains on the table and the pushing efficiency of a push; (e) The evolution of the group cohesion. (i) 74 particles; (ii) 95 particles; (iii) 128 particles; (iv) 140 candies.