ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

TL;DR

ViPlan introduces the first open-source benchmark for visual planning with symbolic predicates and Vision-Language Models, featuring two domains (Blocksworld and household robotics) and two evaluation settings (VLM-as-grounder and VLM-as-planner). The study benchmarks nine open-source VLM families and two closed models, revealing domain-dependent strengths: VLM-grounding excels in abstract, block-based tasks, while direct VLM planning benefits more from commonsense and error recovery in household tasks. Across models, Chain-of-Thought prompting shows limited or no consistent improvement, underscoring current visual reasoning limits. ViPlan provides a standardized evaluation framework to compare VLM-grounded symbolic planning with direct VLM planning and highlights key challenges for grounding accuracy, compounding errors, and robust task planning in vision-guided agents.

Abstract

Integrating Large Language Models with symbolic planners is a promising direction for obtaining verifiable and grounded plans compared to planning in natural language, with recent works extending this idea to visual domains using Vision-Language Models (VLMs). However, rigorous comparison between VLM-grounded symbolic approaches and methods that plan directly with a VLM has been hindered by a lack of common environments, evaluation protocols and model coverage. We introduce ViPlan, the first open-source benchmark for Visual Planning with symbolic predicates and VLMs. ViPlan features a series of increasingly challenging tasks in two domains: a visual variant of the classic Blocksworld planning problem and a simulated household robotics environment. We benchmark nine open-source VLM families across multiple sizes, along with selected closed models, evaluating both VLM-grounded symbolic planning and using the models directly to propose actions. We find symbolic planning to outperform direct VLM planning in Blocksworld, where accurate image grounding is crucial, whereas the opposite is true in the household robotics tasks, where commonsense knowledge and the ability to recover from errors are beneficial. Finally, we show that across most models and methods, there is no significant benefit to using Chain-of-Thought prompting, suggesting that current VLMs still struggle with visual reasoning.

Figures (10)

• Figure 1: Planning with VLMs. Two settings of VLMs used by agents for planning a set of actions to reach a goal. VLM-as-planner uses the VLM directly to produce a new plan after every action. VLM-as-grounder uses the VLM to ground a symbolic agent's plans to the observations from the environment. Grounding takes the form of yes-no question-answering about whether the conditions that make an action executable are met and whether the expected outcomes of the action are realized.
• Figure 2: Compounding Errors in Planning. Analysis of the fractions of tasks solved in ViPlan-BW (all difficulties), based on the number of predictions a model would need to answer correctly to succeed in the VLM-as-grounder evaluation setting. Benchmarks that ask independent questions to VLMs and measure their accuracy do not capture the effect that compounding errors have on solving a problem, and would correspond to measuring only one prediction. Most models score well on the single prediction, but deteriorate quickly as the errors compound.
• Figure 3: Example of the basic components for formal planning with PDDL. A PDDL domain includes the list of possible lifted actions, which are then grounded by a PDDL problem, that provides the initial and goal state. A symbolic planner takes as input the PDDL domain and problem to generate a symbolic plan to reach the desired goal state through a sequence of N action.
• Figure 4: Planning versus Grounding. The VLM-as-grounder approach excels in ViPlan-BW (top), where GPT-4.1 and InternVL3 78B complete a significant fraction of tasks. The VLM-as-planner approach is instead better on ViPlan-HH (bottom), where medium, large and closed models perform generally better than with VLM-as-grounder.
• Figure 5: Impact of Chain-of-Thought. The plots show the difference in success rate when using (top half) and not using (bottom half) zero-shot CoT prompting for all domains and evaluation settings. The rightmost column of each plot reports the average difference across models for each split.