3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding

TL;DR

3DRS investigates how well multimodal large language models capture 3D scene information and demonstrates that higher 3D-awareness correlates with better downstream performance. It introduces a two-pronged approach: a correspondence-based 3D supervision loss and a distillation framework that leverages pretrained 3D foundation models to align MLLM visual features with rich 3D priors. The method yields consistent gains across multiple MLLMs and 3D benchmarks, outperforming specialist and several 3D-generalist baselines, and highlights the superiority of 3D foundation-model supervision over purely 2D cues. This work underscores the importance of integrating explicit 3D priors into MLLMs for robust 3D scene understanding and provides a practical, low-overhead path via offline distilled targets. Overall, 3DRS offers a general, effective strategy to enhance 3D reasoning in multimodal models with broad applicability to grounding, captioning, and QA tasks in real-world 3D environments.

Abstract

Recent advances in scene understanding have leveraged multimodal large language models (MLLMs) for 3D reasoning by capitalizing on their strong 2D pretraining. However, the lack of explicit 3D data during MLLM pretraining limits 3D representation capability. In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal a strong positive correlation between the quality of 3D-aware representation and downstream task performance. Motivated by this, we propose 3DRS, a framework that enhances MLLM 3D representation learning by introducing supervision from pretrained 3D foundation models. Our approach aligns MLLM visual features with rich 3D knowledge distilled from 3D models, effectively improving scene understanding. Extensive experiments across multiple benchmarks and MLLMs -- including visual grounding, captioning, and question answering -- demonstrate consistent performance gains. Project page: https://visual-ai.github.io/3drs

Figures (6)

• Figure 1: Enhancing 3D awareness of MLLMs to improve downstream performance. (a) Besides the common text supervision for MLLMs, 3DRS adopts 3D foundation models to supervise 3D-aware visual representation learning in MLLMs. (b) Combined with 3DRS, we achieve consistent performance improvement across multiple MLLMs and benchmarks.
• Figure 2: Performance across correspondence score quartiles. Model performance across correspondence score quartiles (Q1--Q4, lowest to highest) for each dataset. Samples were divided into quartiles by their correspondence scores. A clear trend is observed: model accuracy improves as the correspondence score increases.
• Figure 3: (a) 3DRS uses a 3D foundation model to supervise the visual representation of the MLLM. (b) 3DRS effectively improves the correspondence learning for MLLMs.
• Figure 4: Visualization of Results Across Different Tasks. (a) Visual Grounding: The predicted bounding box closely aligns with the ground truth. (b) Object Captioning: Our method generates accurate captions for each referred object. (c) Question Answering: The model provides precise answers, where we use the red rectangles to indicate the visual cues utilized for each response. Best viewed when zoomed in.
• Figure 5: Visualization of Results Across Different Tasks. (a) Visual Grounding: The predicted bounding box closely aligns with the ground truth. (b) Object Captioning: Our method generates accurate captions for each referred object. (c) Question Answering: The model provides precise answers, where we use the red rectangles to indicate the visual cues utilized for each response. Best viewed when zoomed in.