DepthLM: Metric Depth From Vision Language Models

TL;DR

DepthLM tackles the challenge that vision-language models lag behind pure-vision models in metric depth estimation from 2D inputs. It introduces visual prompting to provide precise pixel references, and intrinsic-conditioned augmentation to normalize camera focal lengths across datasets, paired with sparse text-based supervised fine-tuning to learn depth without dense heads or regression losses. A 3B DepthLM model delivers more than a $2\times$ gain in accuracy across four indoor/outdoor datasets, approaching the performance of expert pure-vision models and enabling high-quality, metric-scaled point clouds. The framework generalizes to other 3D tasks, offering a unified, scalable approach to multi-task 3D understanding with vision-language models.

Abstract

Vision language models (VLMs) can flexibly address various vision tasks through text interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from 2D inputs. On the other hand, expert pure vision models achieve super-human accuracy in metric depth estimation, a key 3D understanding task. However, they require task-specific architectures and losses. Such difference motivates us to ask: Can VLMs reach expert-level accuracy without architecture or loss change? We take per-pixel metric depth estimation as the representative task and show that the answer is yes! Surprisingly, comprehensive analysis shows that text-based supervised-finetuning with sparse labels is sufficient for VLMs to unlock strong 3D understanding, no dense prediction head or complex regression/regularization loss is needed. The bottleneck for VLMs lies actually in pixel reference and cross-dataset camera ambiguity, which we address through visual prompting and intrinsic-conditioned augmentation. With much smaller models, our method DepthLM surpasses the accuracy of most advanced VLMs by over 2x, making VLMs for the first time comparable with pure vision models. Interestingly, without explicit enforcement during training, VLMs trained with DepthLM naturally avoids over-smoothing, having much fewer flying points at boundary regions than pure vision models. The simplicity of DepthLM also enables a single VLM to cover various 3D tasks beyond metric depth. Our code and model will be released at the link below.

Figures (9)

• Figure 1: We propose DepthLM, a simple and effective method that turns VLMs into strong pixel-level metric depth estimators. The latest VLMs, including GPT-5, still struggle in understanding 3D from 2D inputs. Our model is the first VLM that has comparable accuracy to pure vision models, and can generate point clouds with accurate metric scales.
• Figure 2: Pixel reference.
• Figure 3: SFT vs GRPO. (a) Negative ${\mathcal{L}}_1$ (NegL1) is the best GRPO reward while most common metrics have reasonable accuracy. (b) While SFT and GRPO achieve similar accuracy given the same train dataset size, SFT has much higher per-sample efficiency. We use Argoverse2 dataset for experiments, see Appendix \ref{['appdx:SFT_GRPO']} for cross dataset evaluation, which shows the same trend.
• Figure 4: Mix data training analysis. For (a) and (b), we train on 500K samples on the mixed datasets of DepthLMBench, and report the average accuracy across all evaluation datasets.
• Figure 5: DepthLM. DepthLM first augment the input image to have a unified focal length. Then, it renders visual markers on the image for pixel reference and uses text to interact with VLMs directly.