Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Benefits of Vision Foundation Models for Unsupervised Domain Adaptation

Brunó B. Englert, Fabrizio J. Piva, Tommie Kerssies, Daan de Geus, Gijs Dubbelman

TL;DR

This work investigates whether Vision Foundation Models (VFMs) and Unsupervised Domain Adaptation (UDA) can complement each other to improve semantic segmentation across both in-target and out-of-target domains. The authors introduce VFM-UDA, integrating a ViT-based VFM encoder/decoder with a streamlined masking strategy, built on a MIC-inspired UDA baseline, and conduct extensive ablations on model size, pre-training, and components. Key findings show that VFM-UDA yields better in-target adaptation than current UDA methods while preserving or enhancing out-of-target generalization, with substantial inference speedups (e.g., $8.4\times$ over MIC for ViT-B/14; $3.3\times$ with ViT-L/14 when scaled by $3.6\times$ more parameters). The results establish new standards and baselines for VFM-assisted UDA in semantic segmentation and highlight scalable benefits when combining VFMs with UDA for robust, real-world deployment.

Abstract

Achieving robust generalization across diverse data domains remains a significant challenge in computer vision. This challenge is important in safety-critical applications, where deep-neural-network-based systems must perform reliably under various environmental conditions not seen during training. Our study investigates whether the generalization capabilities of Vision Foundation Models (VFMs) and Unsupervised Domain Adaptation (UDA) methods for the semantic segmentation task are complementary. Results show that combining VFMs with UDA has two main benefits: (a) it allows for better UDA performance while maintaining the out-of-distribution performance of VFMs, and (b) it makes certain time-consuming UDA components redundant, thus enabling significant inference speedups. Specifically, with equivalent model sizes, the resulting VFM-UDA method achieves an 8.4$\times$ speed increase over the prior non-VFM state of the art, while also improving performance by +1.2 mIoU in the UDA setting and by +6.1 mIoU in terms of out-of-distribution generalization. Moreover, when we use a VFM with 3.6$\times$ more parameters, the VFM-UDA approach maintains a 3.3$\times$ speed up, while improving the UDA performance by +3.1 mIoU and the out-of-distribution performance by +10.3 mIoU. These results underscore the significant benefits of combining VFMs with UDA, setting new standards and baselines for Unsupervised Domain Adaptation in semantic segmentation.

Exploring the Benefits of Vision Foundation Models for Unsupervised Domain Adaptation

TL;DR

This work investigates whether Vision Foundation Models (VFMs) and Unsupervised Domain Adaptation (UDA) can complement each other to improve semantic segmentation across both in-target and out-of-target domains. The authors introduce VFM-UDA, integrating a ViT-based VFM encoder/decoder with a streamlined masking strategy, built on a MIC-inspired UDA baseline, and conduct extensive ablations on model size, pre-training, and components. Key findings show that VFM-UDA yields better in-target adaptation than current UDA methods while preserving or enhancing out-of-target generalization, with substantial inference speedups (e.g., over MIC for ViT-B/14; with ViT-L/14 when scaled by more parameters). The results establish new standards and baselines for VFM-assisted UDA in semantic segmentation and highlight scalable benefits when combining VFMs with UDA for robust, real-world deployment.

Abstract

Achieving robust generalization across diverse data domains remains a significant challenge in computer vision. This challenge is important in safety-critical applications, where deep-neural-network-based systems must perform reliably under various environmental conditions not seen during training. Our study investigates whether the generalization capabilities of Vision Foundation Models (VFMs) and Unsupervised Domain Adaptation (UDA) methods for the semantic segmentation task are complementary. Results show that combining VFMs with UDA has two main benefits: (a) it allows for better UDA performance while maintaining the out-of-distribution performance of VFMs, and (b) it makes certain time-consuming UDA components redundant, thus enabling significant inference speedups. Specifically, with equivalent model sizes, the resulting VFM-UDA method achieves an 8.4 speed increase over the prior non-VFM state of the art, while also improving performance by +1.2 mIoU in the UDA setting and by +6.1 mIoU in terms of out-of-distribution generalization. Moreover, when we use a VFM with 3.6 more parameters, the VFM-UDA approach maintains a 3.3 speed up, while improving the UDA performance by +3.1 mIoU and the out-of-distribution performance by +10.3 mIoU. These results underscore the significant benefits of combining VFMs with UDA, setting new standards and baselines for Unsupervised Domain Adaptation in semantic segmentation.
Paper Structure (27 sections, 2 figures)

This paper contains 27 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. VFM-UDA
  5. UDA baseline.
  6. VFM encoder.
  7. VFM decoder.
  8. VFM masking.
  9. Experimental set-up
  10. Domain adaptation setup.
  11. In- and out-of-target evaluation.
  12. Baseline, UDA, and oracle.
  13. Datasets.
  14. Implementation details.
  15. Results
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Generalization capabilities of UDA methods and VFMs. UDA is designed to adapt a model from a labeled source domain to an unlabeled target domain, whereas VFMs capture a broad spectrum of data distributions, contributing to the overall generalization. The goal of this research is to investigate and leverage the combined in- and out-of-target generalization capabilities of UDA and VFMs.
  • Figure 2: Decoder head architecture.