Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs

Qiucheng Wu, Handong Zhao, Michael Saxon, Trung Bui, William Yang Wang, Yang Zhang, Shiyu Chang

TL;DR

<3-5 sentence high-level summary> The paper introduces Visual Spatial Planning (VSP), a benchmark designed to evaluate how vision-language models perceive spatial layouts and formulate action plans in visual scenes. It comprises two scenarios, Maze Navigation and Blocks World, each with four sub-tasks that isolate perception and reasoning, totaling about 4.4k questions with progressive difficulty. Across a range of private and open-source baselines, the study reveals substantial gaps in both perception and reasoning, with perception identified as a primary bottleneck; textual input can mitigate some perception issues, highlighting where models struggle. The authors further show that targeted fine-tuning on open-source models yields improvements and discuss directions for stronger spatial understanding and planning in VLMs, with the benchmark and evaluation scripts publicly released to foster progress.

Abstract

Vision language models (VLMs) are an exciting emerging class of language models (LMs) that have merged classic LM capabilities with those of image processing systems. However, the ways that these capabilities combine are not always intuitive and warrant direct investigation. One understudied capability in VLMs is visual spatial planning -- the ability to comprehend the spatial arrangements of objects and devise action plans to achieve desired outcomes in visual scenes. In our study, we introduce VSP, a benchmark that 1) evaluates the spatial planning capability in these models in general, and 2) breaks down the visual planning task into finer-grained sub-tasks, including perception and reasoning, and measure the LMs capabilities in these sub-tasks. Our evaluation shows that both open-source and private VLMs fail to generate effective plans for even simple spatial planning tasks. Evaluations on the fine-grained analytical tasks further reveal fundamental deficiencies in the models' visual perception and bottlenecks in reasoning abilities, explaining their worse performance in the general spatial planning tasks. Our work illuminates future directions for improving VLMs' abilities in spatial planning. Our benchmark is publicly available at https://github.com/UCSB-NLP-Chang/Visual-Spatial-Planning.

VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs

TL;DR

<3-5 sentence high-level summary> The paper introduces Visual Spatial Planning (VSP), a benchmark designed to evaluate how vision-language models perceive spatial layouts and formulate action plans in visual scenes. It comprises two scenarios, Maze Navigation and Blocks World, each with four sub-tasks that isolate perception and reasoning, totaling about 4.4k questions with progressive difficulty. Across a range of private and open-source baselines, the study reveals substantial gaps in both perception and reasoning, with perception identified as a primary bottleneck; textual input can mitigate some perception issues, highlighting where models struggle. The authors further show that targeted fine-tuning on open-source models yields improvements and discuss directions for stronger spatial understanding and planning in VLMs, with the benchmark and evaluation scripts publicly released to foster progress.

Abstract

Vision language models (VLMs) are an exciting emerging class of language models (LMs) that have merged classic LM capabilities with those of image processing systems. However, the ways that these capabilities combine are not always intuitive and warrant direct investigation. One understudied capability in VLMs is visual spatial planning -- the ability to comprehend the spatial arrangements of objects and devise action plans to achieve desired outcomes in visual scenes. In our study, we introduce VSP, a benchmark that 1) evaluates the spatial planning capability in these models in general, and 2) breaks down the visual planning task into finer-grained sub-tasks, including perception and reasoning, and measure the LMs capabilities in these sub-tasks. Our evaluation shows that both open-source and private VLMs fail to generate effective plans for even simple spatial planning tasks. Evaluations on the fine-grained analytical tasks further reveal fundamental deficiencies in the models' visual perception and bottlenecks in reasoning abilities, explaining their worse performance in the general spatial planning tasks. Our work illuminates future directions for improving VLMs' abilities in spatial planning. Our benchmark is publicly available at https://github.com/UCSB-NLP-Chang/Visual-Spatial-Planning.
Paper Structure (27 sections, 5 figures, 7 tables)

This paper contains 27 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General planning in LMs
  4. Spatial and visual planning in LMs
  5. Benchmarks for VLMs
  6. The Visual Spatial Planning Benchmark
  7. Overview of the Benchmark
  8. The Maze Navigation Scenario
  9. The Blocksword Scenario
  10. Benchmark Creation Process
  11. Experiments
  12. Baselines
  13. Main Task (Spatial Planning) Evaluation
  14. The Perception and Reasoning Sub-tasks Evaluation
  15. The Effects of Visual Input Perception and Reasoning
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Overview of the Maze Navigation scenario.
  • Figure 2: Overview of the Blocks World scenario.
  • Figure 3: Benchmark creation process. Left: We prepare input images that fulfill the task requirements with different difficulties. Mid: We formulate input prompts for each task. The input prompts consists of interleaved texts and images. Right: We develop automatic evaluation process for each task.
  • Figure 4: The visual and corresponding textual inputs.
  • Figure 5: Performance comparison with the visual/textual input. When the environment is described by text instead of image, the performance increases significantly.