Open-World Task and Motion Planning via Vision-Language Model Inferred Constraints

TL;DR

The paper tackles open-world, long-horizon robotic manipulation where goals are described in natural language and cannot be captured by fixed predicate vocabularies. It introduces OWL-TAMP, a framework that uses Vision-Language Models to generate both discrete partial-plan constraints and continuous constraint code (e.g., VLMPose) to augment traditional Task and Motion Planning. By enforcing these language-driven constraints within a TAMP solver, OWL-TAMP preserves planning guarantees while expanding task expressivity, and demonstrates superior performance on simulated open-world tasks and real hardware. This work bridges foundation-model reasoning with robot control, enabling open-world, language-driven manipulation without manual symbolic enlargement of the planner’s vocabulary.

Abstract

Foundation models trained on internet-scale data, such as Vision-Language Models (VLMs), excel at performing a wide variety of common sense tasks like visual question answering. Despite their impressive capabilities, these models cannot currently be directly applied to challenging robot manipulation problems that require complex and precise continuous reasoning over long horizons. Task and Motion Planning (TAMP) systems can control high-dimensional continuous systems over long horizons via a hybrid search over traditional primitive robot skills. However, these systems require detailed models of how the robot can impact its environment, preventing them from directly interpreting and addressing novel human objectives, for example, an arbitrary natural language goal. We propose deploying VLMs within TAMP systems by having them generate discrete and continuous language-parameterized constraints that enable TAMP to reason about open-world concepts. Specifically, we propose algorithms for VLM partial planning that constrain a TAMP system's discrete temporal search and VLM continuous constraints interpretation to augment the traditional manipulation constraints that TAMP systems seek to satisfy. Experiments demonstrate that our approach -- OWL-TAMP -- outperforms several related baselines, including those that solely use TAMP or VLMs for planning, across several long-horizon manipulation tasks specified directly through natural language. We additionally demonstrate that our approach is compatible with a variety of TAMP systems and can be deployed to solve challenging manipulation tasks on real-world hardware.

Figures (6)

• Figure 1: Success rates on all tasks. We report the success rate (i.e., whether the task was successfully accomplished, as measured by manually-written checks), across 10 random seeds. We run a one-tailed z-test with $\alpha = 0.1$ and bold all top-scoring success rates. The final column displays the average success rate across all trials on all tasks.
• Figure 2: Hybrid traditional and VLM constraint satisfaction. The constraint network for a candidate plan to pick and place the apple that achieves the goal "put the orange on the table where the apple initially is" in Figure \ref{['fig:system']}. Due to the Collision and VLMPose constraints, this constraint network isn't satisfiable; however, a plan that first moves apple out of the way before placing apple is.
• Figure 3: Simulated RAVENS-YCB Manipulation Tasks. Intermediate states during execution of our approach on 3 simulated tasks.
• Figure 4: Real-World Tasks. Two tasks from our real-world system deployment (Section \ref{['sec:experiments']}) and their corresponding goal descriptions.
• Figure 5: Soundness rates on select methods and tasks. We present the average 'soundness rate' (i.e., 1 - $\frac{\text{\# false positives}}{\text{\# total trials}}$) of our approach and several relevant baselines averaged across all tasks that require non-trivial discrete or continuous constraint generation. Higher rates indicate that the method has fewer false positives. Table \ref{['tab:soundness_results']} in the appendix shows a more fine-grained breakdown of soundness results.