Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergy

TL;DR

Vid-LLM tackles 3D vision–language reasoning from monocular video by eliminating external 3D data requirements. It integrates a reconstruction branch and a 3D VL reasoning branch through a Cross-Task Adapter, and uses a Metric Depth Model to recover real-scale geometry, aided by a two-stage dual-teacher distillation training regime. The approach achieves state-of-the-art results on 3D QA, dense captioning, and visual grounding among video-based models and competitive performance against joint reconstruction–reasoning frameworks, all with faster inference. This work demonstrates practical, scalable video-based 3D reasoning with robust geometry–semantics interaction for real-world deployment.

Abstract

Recent developments in Multimodal Large Language Models (MLLMs) have significantly improved Vision-Language (VL) reasoning in 2D domains. However, extending these capabilities to 3D scene understanding remains a major challenge. Existing 3D Multimodal Large Language Models (3D-MLLMs) often depend on 3D data inputs, which limits scalability and generalization. To address this limitation, we propose Vid-LLM, a video-based 3D-MLLM that directly processes video inputs without requiring external 3D data, making it practical for real-world deployment. In our method, the geometric prior are directly used to improve the performance of the sceen perception. To integrate the geometric cues into the MLLM compactly, we design a Cross-Task Adapter (CTA) module to align the 3D geometric priors with the vision-language representations. To ensure geometric consistency and integrity, we introduce a Metric Depth Model that recovers real-scale geometry from the reconstruction outputs. Finally, the model is fine-tuned with a two-stage distillation optimization strategy, realizing fast convergence and stabilizes training. Extensive experiments across diverse benchmarks verified the effectiveness of our method on 3D Question Answering, 3D Dense Captioning and 3D Visual Grounding tasks, demonstrating the superior multi-task capabilities.

Figures (8)

• Figure 1: We propose Vid-LLM to achieve diverse 3D vision-language reasoning tasks using only video inputs.
• Figure 2: Architecture of Vid-LLM. From video, a shared DINOv2 encoder produces tokens that are bidirectionally fused by Cross-Task Adapter with learnable Bridge Tokens, yielding geometric and semantic streams. The reconstruction branch predicts camera poses, depth and recovers real-scale via a Metric-Bins module, while the 3D-VL branch lifts features into 3D tokens for LLM reasoning.
• Figure 3: Overview of the two-stage training strategy. Stage-1 employs dual-teacher distillation to align geometry and semantics, and Stage-2 jointly optimizes reconstruction and 3D vision–language tasks.
• Figure 4: Qualitative results of 3D visual grounding on the ScanRefer dataset. The predicted 3D bounding boxes are visualized on point clouds reconstructed by our model.
• Figure 5: Ablation on Cross-Task Adapter (CTA) and Metric Depth (MD) Modules.