Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding

TL;DR

This work proposes Video-3D LLM, a generalist model for 3D scene understanding that treats 3D environments as dynamic videos and fuses depth-derived 3D coordinates via 3D position encoding into video representations. By backprojecting depth to global coordinates, applying sinusoidal 3D-PE, and a multi-task training regime, the model excels at 3D grounding, dense captioning, and QA, achieving state-of-the-art results on ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D while using only a fraction of prior 3D data. A maximum coverage frame-sampling strategy enhances efficiency by selecting informative frames that cover the scene, reducing inference time without sacrificing performance. Overall, the approach demonstrates the viability and practicality of adapting video LLMs to complex 3D spatial reasoning tasks.

Abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has significantly impacted various multimodal tasks. However, these models face challenges in tasks that require spatial understanding within 3D environments. Efforts to enhance MLLMs, such as incorporating point cloud features, have been made, yet a considerable gap remains between the models' learned representations and the inherent complexity of 3D scenes. This discrepancy largely stems from the training of MLLMs on predominantly 2D data, which restricts their effectiveness in comprehending 3D spaces. To address this issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM, for 3D scene understanding. By treating 3D scenes as dynamic videos and incorporating 3D position encoding into these representations, our Video-3D LLM aligns video representations with real-world spatial contexts more accurately. In addition, we have implemented a maximum coverage sampling technique to optimize the trade-off between computational cost and performance. Extensive experiments demonstrate that our model achieves state-of-the-art performance on several 3D scene understanding benchmarks, including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.

Figures (4)

• Figure 1: Comparison of previous work and our method: (a) Previous 3D LLMs are initialized on MLLMs trained solely on image-text pairs, and learn point cloud or voxel representations via fine-tuning on 3D scenes. The 3D point clouds are reconstructed from RGB-D videos. (b) Our method directly utilizes video frames and 3D coordinates as input, where the 3D coordinates are converted from depths through coordinate transformation. We then transfer the ability of video understanding to 3D scene understanding by injecting position information into video representations.
• Figure 2: The overview of the model architecture. (a) shows the integration of video sequence and global coordinates for creating position-aware video representations. (b) and (c) detail the examples of 3D dense captioning and 3D visual grounding, respectively. Our approach can generalize well to other 3D tasks.
• Figure 3: The visualization results on ScanRefer. The green/red/blue colors indicate the correct/incorrect/ground truth boxes.
• Figure 4: The visualization results on Scan2Cap. The input boxes are marked in blue.