I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction

TL;DR

The paper tackles the challenge of visual spatial reasoning in Visual Language Models by introducing ZeroVLM, which leverages 3D viewpoint reconstruction via Zero-1-to-3 and a view-prompting mechanism to access richer spatial information. By generating left, right, and random views and stitching them with the original image, ZeroVLM enhances spatial relation understanding when paired with VLM backbones like LLaVA or MiniGPT-4, with a dedicated view prompt guiding reasoning. Across four visual spatial reasoning datasets, the approach achieves up to $19.48\%$ accuracy improvement, though single-view generally outperforms multi-view configurations due to complexity. The work highlights the value of 3D viewpoint synthesis and textual prompts for improving VLM reasoning, while also outlining limitations and potential risks related to data, compute, and deployment ethics.

Abstract

Visual Language Models (VLMs) are essential for various tasks, particularly visual reasoning tasks, due to their robust multi-modal information integration, visual reasoning capabilities, and contextual awareness. However, existing \VLMs{}' visual spatial reasoning capabilities are often inadequate, struggling even with basic tasks such as distinguishing left from right. To address this, we propose the \ours{} model, designed to enhance the visual spatial reasoning abilities of VLMS. ZeroVLM employs Zero-1-to-3, a 3D reconstruction model for obtaining different views of the input images and incorporates a prompting mechanism to further improve visual spatial reasoning. Experimental results on four visual spatial reasoning datasets show that our \ours{} achieves up to 19.48% accuracy improvement, which indicates the effectiveness of the 3D reconstruction and prompting mechanisms of our ZeroVLM.

Figures (10)

• Figure 1: An example of the VQA task, where humans can easily recognize positions under different views, but the vanilla LLaVA liu2024visual can only predict correctly from certain views. By performing 3D reconstruction to obtain different views of the image, we can improve LLaVA's predictive accuracy.
• Figure 2: Single-view images were generated using Zero-1-to-3 to produce left-view, right-view, and random-view images. Multi-view images were created by combining these different single-view images in various configurations.
• Figure 3: An overview of our proposed ZeroVLM model. Our ZeroVLM first uses Zero-1-to-3 to perform 3D reconstruction to obtain different views of the input image, and then it stitches the original images with these different views to obtain the stitched image, which is the input of a VLM for answer prediction.
• Figure 4: These view prompts are manually constructed by us. View prompt comparison between single-view images and multi-view images. {question} is the corresponding question in the prompt.
• Figure 5: For both single-view and multi-view datasets, we employed a view prompt. The aim was to explore whether perspective language models could enhance their visual spatial reasoning abilities through improvements at the textual level.