KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulation

TL;DR

KUDA addresses open-vocabulary robotic manipulation by unifying vision-language prompting with data-driven dynamics through a keypoint-based intermediate representation. A VLM generates keypoint target specifications from language and RGBD observations, which are translated into 3D cost functions for a neural dynamics model to optimize robot trajectories via MPPI in a closed loop. The system uses a Top-K prompt library and a CLIP-based retriever to provide few-shot prompts within token limits, enabling robust generalization to diverse objects and materials. KUDA demonstrates state-of-the-art performance on tasks across ropes, granular materials, and deformable objects, highlighting the practical potential of combining language-grounded aim with learned dynamics for flexible manipulation.

Abstract

With the rapid advancement of large language models (LLMs) and vision-language models (VLMs), significant progress has been made in developing open-vocabulary robotic manipulation systems. However, many existing approaches overlook the importance of object dynamics, limiting their applicability to more complex, dynamic tasks. In this work, we introduce KUDA, an open-vocabulary manipulation system that integrates dynamics learning and visual prompting through keypoints, leveraging both VLMs and learning-based neural dynamics models. Our key insight is that a keypoint-based target specification is simultaneously interpretable by VLMs and can be efficiently translated into cost functions for model-based planning. Given language instructions and visual observations, KUDA first assigns keypoints to the RGB image and queries the VLM to generate target specifications. These abstract keypoint-based representations are then converted into cost functions, which are optimized using a learned dynamics model to produce robotic trajectories. We evaluate KUDA on a range of manipulation tasks, including free-form language instructions across diverse object categories, multi-object interactions, and deformable or granular objects, demonstrating the effectiveness of our framework. The project page is available at http://kuda-dynamics.github.io.

Figures (4)

• Figure 1: KUDA is an open-vocabulary manipulation system that uses keypoints to unify the visual prompting of vision language models (VLMs) and dynamics modeling. Taking the RGBD observation and the language instruction as inputs, KUDA samples keypoints in the environment, then uses a VLM to generate code specifying keypoint-based target specification. These keypoints are translated into a cost function for model-based planning with learned dynamics models, enabling open-vocabulary manipulation across various object categories.
• Figure 2: Overview of KUDA. Taking the RGBD observations and a language instruction as inputs, we first utilize the large vision model to obtain the keypoints and label them on the RGB image to obtain the visual prompt (green dot C marks the center reference point). Next, the vision-language model generates code for target specifications, which are projected into 3D space to construct the 3D objectives. Lastly, we utilize the pre-trained dynamics model for model-based planning. After a certain number of actions, the VLM is re-queried with the current observation, enabling high-level closed-loop planning to correct VLM and execution errors.
• Figure 3: Qualitative Results of the Rollouts. We show the target specification and robot executions of various tasks on different objects, highlight the effectiveness of our framework. We show the initial state and the target specification visualization of our system, along with the robot executions, to demonstrate the performance of our framework on various manipulation tasks. Note that we show the granular collection task to exhibit how our VLM-level closed-loop control works in our two VLM-level loops.
• Figure 4: Visualizations of Error Breakdown. We provide a detailed breakdown of each failure mode, marked in red. While we achieved an $80\%$ success rate across 60 trials for various tasks, the primary cause of failure was perception errors, accounting for $10\%$ of all trials and $50\%$ of the failure cases.