Physics-Constrained Robot Grasp Planning for Dynamic Tool Use

TL;DR

The paper addresses robust robot tool use under dynamic and cluttered environments by introducing iTuP, a framework that fuses open-vocabulary semantic grounding with physics-constrained grasp generation and short-horizon trajectory planning. A key component, SDG-Net, predicts trajectory-conditioned penalties for induced torque, slip, and alignment to score grasps via a physics-based objective $C$, while a two-level VLM grounding module ties task semantics to actionable contacts. Across simulation and real-world experiments (hammering, sweeping, knocking, reaching), iTuP outperforms geometry-only and VLM-only baselines, demonstrating improved torque minimization, slip resistance, and tool–target alignment, leading to higher task success rates in quasi-static, dynamic, and cluttered conditions. These results establish physics-constrained grasping as essential for reliable tool use, and highlight the value of integrating semantic grounding with dynamic stability considerations for generalizable manipulation in unstructured settings.

Abstract

Tool use requires not only selecting appropriate tools but also generating grasps and motions that remain stable under dynamic interactions. Existing approaches largely focus on high-level tool grounding or quasi-static manipulation, overlooking stability in dynamic and cluttered regimes. We introduce iTuP (inverse Tool-use Planning), a framework that outputs robot grasps explicitly tailored for tool use. iTuP integrates a physics-constrained grasp generator with a task-conditional scoring function to produce grasps that remain stable during dynamic tool interactions. These grasps account for manipulation trajectories, torque requirements, and slip prevention, enabling reliable execution of real-world tasks. Experiments across hammering, sweeping, knocking, and reaching tasks demonstrate that iTuP outperforms geometry-based and vision-language model (VLM)-based baselines in grasp stability and task success. Our results underscore that physics-constrained grasping is essential for robust robot tool use in quasi-static, dynamic, and cluttered environments.

Figures (5)

• Figure 1: Tool use tasks could involve high-torque interactions. Unstable grasps can result in gripper-tool drift, leading to failures. Our proposed framework iTuP enables task-appropriate stable grasping for robust tool manipulation.
• Figure 2: Physics-Grounded Grasping. Given grounded contacts and a short, task-conditioned interaction trajectory, the generator scores grasp candidates using trajectory-conditioned penalties for (a) induced interaction torque, (b) slip margin, and (c) normal alignment. A multimodal penalty network (SDG-Net) learns these penalties and returns the grasp with the minimal torque penalty.
• Figure 3: VLM-driven Hierarchical Tool Grounding. We apply a two-level granularity tool grounding framework. In the coarse grounding stage, given a visual observation and a user instruction (e.g., "Hammer the nail"), the Set-of-Mark module segments and indexes objects, then uses a VLM to retrieve a selection of objects and tools. In the fine grounding stage, a sampler generates candidate contact points on the selected tool and object, and a VLM is employed to select the best contact configuration. The red arrows realize the VLM-provided directions as normals, while the blue arrow on the object represents the scalar follow-through distance along this normal.
• Figure 4: Failure case for a reach task. iTuP without SDG-Net fails to provide a stable grasp against the torque involved in knocking down blocks out of range.
• Figure 5: Visualization of successful real-world trials. See Fig. \ref{['fig:title']} for a successful reach task.