Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding
Xuefei Sun, Doncey Albin, Cecilia Mauceri, Dusty Woods, Christoffer Heckman
TL;DR
This work tackles the challenge of learning precise spatial referring expressions in multimodal models by introducing SUN-Spot v2.0, an RGB-D dataset with landmark annotations, and Set-of-Marks prompting to align visual regions with caption mentions. Spatial-LLaVA, a fine-tuned MLLM built on a frozen vision encoder and trained with 75k GPT-generated spatial QA conversations, achieves a 3.15% zero-shot improvement in Visual Spatial Reasoning and sets new state-of-the-art results on SUN-Spot v2.0 Expert, SUNRefer, and VSR benchmarks. The approach emphasizes grounding accuracy over object semantics, enabling robust spatial reasoning useful for autonomous navigation and human-robot interaction. The paper also provides a scalable data-generation pipeline via SoM and GPT-based conversation synthesis that can be extended to broader visual-grounding tasks and robotic applications.
Abstract
Multimodal large language models (MLLMs) have demonstrated remarkable abilities in comprehending visual input alongside text input. Typically, these models are trained on extensive data sourced from the internet, which are sufficient for general tasks such as scene understanding and question answering. However, they often underperform on specialized tasks where online data is scarce, such as determining spatial relationships between objects or localizing unique target objects within a group of objects sharing similar features. In response to this challenge, we introduce the SUN-Spot v2.0 dataset1, now comprising a total of 90k image-caption pairs and additional annotations on the landmark objects. Each image-caption pair utilizes Set-of-Marks prompting as an additional indicator, mapping each landmark object in the image to the corresponding object mentioned in the caption. Furthermore, we present Spatial-LLaVA, an MLLM trained on conversational data generated by a state-of-the-art language model using the SUNSpot v2.0 dataset. Our approach ensures a robust alignment between the objects in the images and their corresponding object mentions in the captions, enabling our model to learn spatial referring expressions without bias from the semantic information of the objects. Spatial-LLaVA outperforms previous methods by 3.15% on the zero-shot Visual Spatial Reasoning benchmark dataset. Spatial-LLaVA is specifically designed to precisely understand spatial referring expressions, making it highly applicable for tasks in real-world scenarios such as autonomous navigation and interactive robotics, where precise object recognition is critical.