The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models?

TL;DR

This paper questions whether 3D point-cloud input truly enhances spatial reasoning in large language models, and it decouples this from generic perception by comparing text-only, vision-only, and multi-modal inputs. It introduces ScanReQA, a benchmark to evaluate forward and backward binary spatial relationships as well as absolute spatial coordinates, and performs a comprehensive evaluation of LLMs, VLMs, and 3D LLMs across three 3D QA benchmarks. The results show that point clouds offer limited benefit, with many models achieving competitive results using text or vision alone, and 3D LLMs struggle to reason about binary spatial relations or leverage 3D coordinates. The findings challenge assumptions about modality advantages in 3D reasoning and provide a dataset and methodological framework to guide future work in multi-modal 3D reasoning.

Abstract

3D Large Language Models (LLMs) leveraging spatial information in point clouds for 3D spatial reasoning attract great attention. Despite some promising results, the role of point clouds in 3D spatial reasoning remains under-explored. In this work, we comprehensively evaluate and analyze these models to answer the research question: \textit{Does point cloud truly boost the spatial reasoning capacities of 3D LLMs?} We first evaluate the spatial reasoning capacity of LLMs with different input modalities by replacing the point cloud with the visual and text counterparts. We then propose a novel 3D QA (Question-answering) benchmark, ScanReQA, that comprehensively evaluates models' understanding of binary spatial relationships. Our findings reveal several critical insights: 1) LLMs without point input could even achieve competitive performance even in a zero-shot manner; 2) existing 3D LLMs struggle to comprehend the binary spatial relationships; 3) 3D LLMs exhibit limitations in exploiting the structural coordinates in point clouds for fine-grained spatial reasoning. We think these conclusions can help the next step of 3D LLMs and also offer insights for foundation models in other modalities. We release datasets and reproducible codes in the anonymous project page: https://3d-llm.xyz.

Figures (7)

• Figure 1: The overview of our evaluation framework on 3D LLMs. In multi-modal input evaluation, we convert the point cloud into visual and textual counterparts and feed them into different models, respectively. In spatial reasoning evaluation, 3D LLMs are required to perform both forward and backward spatial reasoning while also inferring the location of the referenced object.
• Figure 2: Visualization of accuracy and attentions scores of LEO.
• Figure 3: The multi-modal data generation pipeline. The 3D scene scan is projected into continuous scene frames which are further uniformly downsampled to $N$ frames as the input for VLMs. We also select frames containing scene objects with object point clouds and leverage caption models to generate a text description of the scene as the input for LLMs.
• Figure 4: Performance overview on ScanQA, SQA3D, and ScanReQA with different modality input.
• Figure 5: Ablation studies with different modality inputs.