VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models

TL;DR

VESSA tackles label-free domain adaptation for vision foundation models by leveraging short object-centric videos in a self-distillation framework. It introduces a frame-pair video-based fine-tuning pipeline with frame sampling, uncertainty-weighted self-distillation, and LoRA-based parameter-efficient updates, coupled with a staged unfreezing strategy to minimize forgetting. Across MVImageNet and CO3D, and using DINO, DINOv2, and TIPS backbones, VESSA yields consistent improvements over base pretraining and image-only baselines, while remaining computationally efficient. This work demonstrates the value of temporal, object-centric supervision for unsupervised domain adaptation and provides publicly available code for practical adoption.

Abstract

Foundation models have advanced computer vision by enabling strong performance across diverse tasks through large-scale pretraining and supervised fine-tuning. However, they may underperform in domains with distribution shifts and scarce labels, where supervised fine-tuning may be infeasible. While continued self-supervised learning for model adaptation is common for generative language models, this strategy has not proven effective for vision-centric encoder models. To address this challenge, we introduce a novel formulation of self-supervised fine-tuning for vision foundation models, where the model is adapted to a new domain without requiring annotations, leveraging only short multi-view object-centric videos. Our method is referred to as VESSA: Video-based objEct-centric Self-Supervised Adaptation for visual foundation models. VESSA's training technique is based on a self-distillation paradigm, where it is critical to carefully tune prediction heads and deploy parameter-efficient adaptation techniques - otherwise, the model may quickly forget its pretrained knowledge and reach a degraded state. VESSA benefits significantly from multi-view object observations sourced from different frames in an object-centric video, efficiently learning robustness to varied capture conditions, without the need of annotations. Through comprehensive experiments with 3 vision foundation models on 2 datasets, VESSA demonstrates consistent improvements in downstream classification tasks, compared to the base models and previous adaptation methods. Code is publicly available at https://github.com/jesimonbarreto/VESSA.

Figures (4)

• Figure 1: We present VESSA, a novel and efficient method for adapting vision foundation models using self-supervised fine-tuning with videos. Starting from a pretrained foundation model applied to a classification problem in a target domain, VESSA adapts the model without using labels by leveraging simple, object-centric videos. The resulting model learns improved representations that better structure the feature space in the target domain, boosting downstream classification accuracy.
• Figure 2: The proposed training pipeline. The model input consists of videos, which first undergo a Frame Selection stage, where $n$ pairs of frames are sampled from each video. These pairs are then passed through the Preprocessing and Augmentation stage, where distinct transformations are applied to the first and second images of each pair. The resulting views are fed into teacher and student networks, both initialized from the same foundation model; LoRA is applied to their architectures for parameter-efficient fine-tuning. Finally, the uncertainty-weighted self-distillation loss (UWSD) is applied to align their representations.
• Figure 3: Qualitative examples of nearest neighbor retrieval ($k = 1$) on the CO3D validation set. We selected some of the most challenging validation samples and retrieved the nearest neighbors using two methods, shown row-wise: the first row displays the query (validation) images; the second row presents the retrievals using raw DINOv2 features; the third row shows the retrievals produced by our method. Images with red borders indicate incorrect retrievals, while green-bordered images represent correct ones. These examples illustrate the effectiveness of leveraging multi-frame video information for representation learning. Notably, our method demonstrates greater focus on the object of interest, whereas the baseline often retrieves matches dominated by background similarity.
• Figure 4: Example frames from the MVImageNet dataset illustrating the differences between global crop input pairs used for the teacher and student networks during training with DINO and VESSA (ours). Our method, VESSA, introduces substantially greater variability in the appearance of the evaluated object. The temporal distance between the selected frames is $\delta = 5$ frames. The first image of each pair shows the global crop from the transformation of view 1, and the second image of each pair shows the global crop corresponding to the transformations of view 2.