Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling
William Xie, Max Conway, Yutong Zhang, Nikolaus Correll
TL;DR
This work shows that vision-language models can be steered to reason about forces and execute force-based manipulation by overlaying object-centric coordinate frames and eliciting wrenches instead of trajectories. A two-step prompting scheme—first spatial reasoning, then physical reasoning—coupled with coordinate-frame labeling enables zero-shot generalization to several contact-rich tasks across multiple robot platforms, achieving roughly half-success in a broad setting. The approach also demonstrates that feedback (robot- or human-provided) can improve task completion, and raises important safety concerns by showing that visual grounding and embodied reasoning can bypass model safeguards. The authors propose a scalable “data flywheel” paradigm to collect data and refine VLM-based manipulation, while acknowledging ethical and safety challenges that demand further safeguards and responsible deployment considerations.
Abstract
Vision language models (VLMs) exhibit vast knowledge of the physical world, including intuition of physical and spatial properties, affordances, and motion. With fine-tuning, VLMs can also natively produce robot trajectories. We demonstrate that eliciting wrenches, not trajectories, allows VLMs to explicitly reason about forces and leads to zero-shot generalization in a series of manipulation tasks without pretraining. We achieve this by overlaying a consistent visual representation of relevant coordinate frames on robot-attached camera images to augment our query. First, we show how this addition enables a versatile motion control framework evaluated across four tasks (opening and closing a lid, pushing a cup or chair) spanning prismatic and rotational motion, an order of force and position magnitude, different camera perspectives, annotation schemes, and two robot platforms over 220 experiments, resulting in 51% success across the four tasks. Then, we demonstrate that the proposed framework enables VLMs to continually reason about interaction feedback to recover from task failure or incompletion, with and without human supervision. Finally, we observe that prompting schemes with visual annotation and embodied reasoning can bypass VLM safeguards. We characterize prompt component contribution to harmful behavior elicitation and discuss its implications for developing embodied reasoning. Our code, videos, and data are available at: https://scalingforce.github.io/.