Probing the 3D Awareness of Visual Foundation Models

TL;DR

This work probes whether visual foundation models trained on 2D data acquire 3D awareness by evaluating frozen representations with dense, multiscale probes on depth and surface normals, and by measuring cross-view geometric correspondence. It compares a broad set of models with varied supervision (self-supervised, supervised, and text-conditioned generation) across indoor scenes and objects using NYUv2, NAVI, ScanNet, SPair, and related datasets. The findings reveal that many models encode visible-surface geometry, with self-supervised and diffusion-based models performing best on single-view depth and normals, while vision-language models lag; multiview consistency remains weak, especially under large viewpoint changes, suggesting view-dependent rather than truly 3D-consistent representations. These results highlight the need for targeted 3D benchmarks and careful consideration of training objectives when aiming for holistic 3D understanding in visual foundation models.

Abstract

Recent advances in large-scale pretraining have yielded visual foundation models with strong capabilities. Not only can recent models generalize to arbitrary images for their training task, their intermediate representations are useful for other visual tasks such as detection and segmentation. Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure? In this work, we analyze the 3D awareness of visual foundation models. We posit that 3D awareness implies that representations (1) encode the 3D structure of the scene and (2) consistently represent the surface across views. We conduct a series of experiments using task-specific probes and zero-shot inference procedures on frozen features. Our experiments reveal several limitations of the current models. Our code and analysis can be found at https://github.com/mbanani/probe3d.

Figures (9)

• Figure 1: Are current visual foundation models 3D aware? We probe the 3D awareness of the learned representations by evaluating their ability to encode the 3D structure of the visible surface and their consistency across views.
• Figure 2: Depth Estimation Results. While pretrained representations exhibit large variation in their ability to represent depth, their performance is consistent on objects and scenes. CLIP and MAE features do not encode depth and appear to instead capture rough priors such as "floor pixels are close". Most models appear to capture the rough structure of the scene and vary in the degree to which they capture details. DINOv2 performs best and accurately captures fine details; e.g., cow's ear, desk chair, and coffee table.
• Figure 3: Surface Normal Qualitative Examples. With the exception of CLIP, models can capture the rough orientation of object and scene surfaces; e.g., floors, walls, ceilings. The main distinction seems to be in how well they capture finer details. Similarly to depth results, we find that DINOv2 and StableDiffusion perform best and can capture fine details such as the edges of the toy car and the white seat. Surprisingly, we find that SAM's predictions are not as detailed despite its ability to predict accurate segmentation boundaries.
• Figure 4: Single view performance correlation. Depth and surface normal performance is highly correlated across domains.
• Figure 5: Correspondence Estimation Qualitative Results. We observe that models can estimate accurate correspondence for small viewpoint changes, but struggle with large viewpoint changes. This is true even if the change is an in-plane rotation as shown with the eagle. This pattern is consistent for both objects and scenes, although performance is not well correlated: SAM and StableDiffusion perform better for scenes, while DeiT and DINOv2 are more consistent for objects. Correspondence color-coded for accuracy.