Does DINOv3 Set a New Medical Vision Standard?

TL;DR

<3-5 sentence high-level summary> This study probes whether DINOv3, a large-scale self-supervised vision transformer trained on natural images, can serve as a universal encoder for medical imaging without domain-specific pre-training. Through a comprehensive benchmark spanning 2D/3D classification and 3D segmentation across X-ray, CT, MRI, PET, WSI, and EM data, the authors examine performance trends as model size and input resolution vary. They find that DINOv3 can set a strong baseline and even surpass some medical foundation models on certain tasks (notably 2D X-ray and 3D CT classification), but exhibits significant limitations in highly domain-shifted modalities (WSI, EM, PET) and shows inconsistent scaling behavior. The work suggests leveraging DINOv3 features as robust priors while pursuing targeted fine-tuning and 2D-to-3D adapters for dense predictions in medical imaging.</p>

Abstract

The advent of large-scale vision foundation models, pre-trained on diverse natural images, has marked a paradigm shift in computer vision. However, how the frontier vision foundation models' efficacies transfer to specialized domains remains such as medical imaging remains an open question. This report investigates whether DINOv3, a state-of-the-art self-supervised vision transformer (ViT) that features strong capability in dense prediction tasks, can directly serve as a powerful, unified encoder for medical vision tasks without domain-specific pre-training. To answer this, we benchmark DINOv3 across common medical vision tasks, including 2D/3D classification and segmentation on a wide range of medical imaging modalities. We systematically analyze its scalability by varying model sizes and input image resolutions. Our findings reveal that DINOv3 shows impressive performance and establishes a formidable new baseline. Remarkably, it can even outperform medical-specific foundation models like BiomedCLIP and CT-Net on several tasks, despite being trained solely on natural images. However, we identify clear limitations: The model's features degrade in scenarios requiring deep domain specialization, such as in Whole-Slide Pathological Images (WSIs), Electron Microscopy (EM), and Positron Emission Tomography (PET). Furthermore, we observe that DINOv3 does not consistently obey scaling law in the medical domain; performance does not reliably increase with larger models or finer feature resolutions, showing diverse scaling behaviors across tasks. Ultimately, our work establishes DINOv3 as a strong baseline, whose powerful visual features can serve as a robust prior for multiple complex medical tasks. This opens promising future directions, such as leveraging its features to enforce multiview consistency in 3D reconstruction.

Figures (5)

• Figure 1: Scaling behavior of DINOv3 models across datasets. The results reveal a non-trivial relationship between performance, model size, and input resolution, where larger models or higher resolutions do not consistently yield better outcomes.
• Figure 2: Cross-domain generalization on Camelyon16 Camelyon16 and Camelyon17 Camelyon17: In-domain vs. Out-of-domain AUC and ACC comparisons.
• Figure 3: Performance comparison across ALN metastasis and receptor status tasks on the BCNB BCNB dataset. The default feature aggregator for the whole-slide images is the attention-based multiple instance learning method ilse2018attention.
• Figure 4: Visualization of a slice from the AC3/4 AC34 dataset and feature embeddings. (a) Raw EM image. (b–d) Feature embeddings extracted from DINOv3-S/16 (b), DINOv3-B/16 (c), and DINOv3-L/16 (d) models, visualized by projecting the first three principal components into RGB space. (e) Corresponding affinity map derived from the raw image.
• Figure 5: Visualization of the first three principal components derived from PCA on image patches. (a) CT images and (b) PET images are shown with their respective PCA visualizations, where each of the first three components is mapped to a color channel. (c) The resulting tumor region can be isolated by thresholding the first principal component to remove the background.