Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors

Duo Zheng, Shijia Huang, Yanyang Li, Liwei Wang

TL;DR

The work tackles the challenge of deriving 3D world understanding from video without explicit 3D inputs. It introduces VG LLM, which incorporates a 3D visual geometry encoder to supply latent geometric priors that augment 2D visual tokens, enabling 3D grounding, dense captioning, and video-object detection within an LLM framework. Through multi-task training and instruction tuning on spatial-reasoning datasets, VG LLM achieves state-of-the-art performance on VSI-Bench and strong generalization across data sources, often surpassing 3D-input-based baselines. The findings demonstrate that explicit 3D geometry priors learned from video can significantly enhance 3D spatial reasoning while maintaining broad multimodal capabilities, reducing reliance on dense 3D sensors.

Abstract

Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them as videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MLLMs to understand and reason in 3D spaces directly from video data, without the need for additional 3D input. We propose a novel and efficient method called the Video-3D Geometry Large Language Model (VG LLM). Our approach utilizes a 3D visual geometry encoder to extract 3D prior information from video sequences. This information is then integrated with visual tokens and input into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the VSI-Bench evaluations.

Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors

TL;DR

The work tackles the challenge of deriving 3D world understanding from video without explicit 3D inputs. It introduces VG LLM, which incorporates a 3D visual geometry encoder to supply latent geometric priors that augment 2D visual tokens, enabling 3D grounding, dense captioning, and video-object detection within an LLM framework. Through multi-task training and instruction tuning on spatial-reasoning datasets, VG LLM achieves state-of-the-art performance on VSI-Bench and strong generalization across data sources, often surpassing 3D-input-based baselines. The findings demonstrate that explicit 3D geometry priors learned from video can significantly enhance 3D spatial reasoning while maintaining broad multimodal capabilities, reducing reliance on dense 3D sensors.

Abstract

Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them as videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MLLMs to understand and reason in 3D spaces directly from video data, without the need for additional 3D input. We propose a novel and efficient method called the Video-3D Geometry Large Language Model (VG LLM). Our approach utilizes a 3D visual geometry encoder to extract 3D prior information from video sequences. This information is then integrated with visual tokens and input into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the VSI-Bench evaluations.

Paper Structure

This paper contains 64 sections, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models (MLLMs).
  4. 3D Large Language Models.
  5. Spatial Reasoning.
  6. Method
  7. Architecture
  8. Preliminary.
  9. 3D Visual Geometry Encoder.
  10. Visual Feature Fusion.
  11. Training
  12. Applying VG LLM to 3D Scene Understanding
  13. Coordinate System and Representation.
  14. 3D Visual Grounding.
  15. 3D Dense Captioning.
  16. ...and 49 more sections

Figures (5)

  • Figure 1: The architecture of our VG LLM. The 3D visual geometry encoder processes a sequence of images to produce globally geometry-aware visual features, while the 2D visual encoder extracts semantic-aware visual features from each individual image. vividly shows that the latent 3D geometry tokens are able to recover the 3D scene if with a dense prediction head wang2025vggt.
  • Figure 2: Qualitative results for 3D visual grounding. The ground truth and prediction are masked in blue and green, respectively. The predicted boxes are directly generated by our model without the refinement process.
  • Figure 3: Qualitative results for 3D video object detection.
  • Figure 4: Visualization of 3D visual grounding. The first three examples are positive, whereas the last two are negative cases.
  • Figure 5: Visualization of 3D video object detection results.