Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anchor-based Robust Finetuning of Vision-Language Models

Jinwei Han, Zhiwen Lin, Zhongyisun Sun, Yingguo Gao, Ke Yan, Shouhong Ding, Yuan Gao, Gui-Song Xia

TL;DR

This work tackles finetuning CLIP-like vision-language models without sacrificing out-of-distribution generalization across domain shifts and zero-shot scenarios. It introduces Anchor-based Robust Finetuning (ARF), which regularizes finetuning with two types of auxiliary semantic anchors: text-compensated anchors generated from captions and image-text anchors retrieved from a CLIP-like candidate pool. The combined training objective adds caption-based and retrieved-anchor contrastive losses to the standard CLIP loss, preserving the original feature space and enhancing OOD robustness. Experiments across domain-shift and zero-shot benchmarks demonstrate state-of-the-art performance while maintaining strong in-distribution accuracy, with ablations confirming the complementary benefits of both anchors.

Abstract

We aim at finetuning a vision-language model without hurting its out-of-distribution (OOD) generalization. We address two types of OOD generalization, i.e., i) domain shift such as natural to sketch images, and ii) zero-shot capability to recognize the category that was not contained in the finetune data. Arguably, the diminished OOD generalization after finetuning stems from the excessively simplified finetuning target, which only provides the class information, such as ``a photo of a [CLASS]''. This is distinct from the process in that CLIP was pretrained, where there is abundant text supervision with rich semantic information. Therefore, we propose to compensate for the finetune process using auxiliary supervision with rich semantic information, which acts as anchors to preserve the OOD generalization. Specifically, two types of anchors are elaborated in our method, including i) text-compensated anchor which uses the images from the finetune set but enriches the text supervision from a pretrained captioner, ii) image-text-pair anchor which is retrieved from the dataset similar to pretraining data of CLIP according to the downstream task, associating with the original CLIP text with rich semantics. Those anchors are utilized as auxiliary semantic information to maintain the original feature space of CLIP, thereby preserving the OOD generalization capabilities. Comprehensive experiments demonstrate that our method achieves in-distribution performance akin to conventional finetuning while attaining new state-of-the-art results on domain shift and zero-shot learning benchmarks.

Anchor-based Robust Finetuning of Vision-Language Models

TL;DR

This work tackles finetuning CLIP-like vision-language models without sacrificing out-of-distribution generalization across domain shifts and zero-shot scenarios. It introduces Anchor-based Robust Finetuning (ARF), which regularizes finetuning with two types of auxiliary semantic anchors: text-compensated anchors generated from captions and image-text anchors retrieved from a CLIP-like candidate pool. The combined training objective adds caption-based and retrieved-anchor contrastive losses to the standard CLIP loss, preserving the original feature space and enhancing OOD robustness. Experiments across domain-shift and zero-shot benchmarks demonstrate state-of-the-art performance while maintaining strong in-distribution accuracy, with ablations confirming the complementary benefits of both anchors.

Abstract

We aim at finetuning a vision-language model without hurting its out-of-distribution (OOD) generalization. We address two types of OOD generalization, i.e., i) domain shift such as natural to sketch images, and ii) zero-shot capability to recognize the category that was not contained in the finetune data. Arguably, the diminished OOD generalization after finetuning stems from the excessively simplified finetuning target, which only provides the class information, such as ``a photo of a [CLASS]''. This is distinct from the process in that CLIP was pretrained, where there is abundant text supervision with rich semantic information. Therefore, we propose to compensate for the finetune process using auxiliary supervision with rich semantic information, which acts as anchors to preserve the OOD generalization. Specifically, two types of anchors are elaborated in our method, including i) text-compensated anchor which uses the images from the finetune set but enriches the text supervision from a pretrained captioner, ii) image-text-pair anchor which is retrieved from the dataset similar to pretraining data of CLIP according to the downstream task, associating with the original CLIP text with rich semantics. Those anchors are utilized as auxiliary semantic information to maintain the original feature space of CLIP, thereby preserving the OOD generalization capabilities. Comprehensive experiments demonstrate that our method achieves in-distribution performance akin to conventional finetuning while attaining new state-of-the-art results on domain shift and zero-shot learning benchmarks.
Paper Structure (18 sections, 5 equations, 5 figures, 4 tables)

This paper contains 18 sections, 5 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Contrastive Learning
  4. Finetuning for Generalization
  5. Preliminaries
  6. Summary of Notations
  7. Contrastive Vision-Language Models
  8. Method
  9. Problem Setup
  10. Anchor-based Robust Finetuning
  11. Text-Compensated Anchor Generation
  12. Image-Text Anchor Retrieval
  13. Experiments
  14. Evaluation Under Domain Shift
  15. Evaluation Under Zero-shot Learning
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Motivation illustration. (a) The out-of-distribution generalization (i.e., domain shift and zero-shot learning) capabilities of CLIP degrade significantly after finetuning on downstream tasks. (b) Images with generated texts and retrieved image-text pairs serve as two types of anchors to regularize the finetuning process of CLIP with auxiliary semantic information.
  • Figure 2: The pipeline of our proposed Anchor-based Robust Finetuning (ARF) comprises a Text-Compensated Anchor Generation (TCAG) module and an Image-Text Anchor Retrieval (ITAR) module. TCAG generates a caption for each image in the finetuning dataset utilizing a pretrained captioner as a text-compensated anchor with rich semantics. ITAR searches for image-text pairs from a candidate set similar to the data on which CLIP was pretrained, ensuring the presence of rich semantics in the image-text-pair anchor. We retrieve those samples related to our downstream tasks. A contrastive loss function, as used in CLIP, is employed for image-text alignment.
  • Figure 3: The pipeline of our Image-Text Anchor Retrieval (ITAR) module. We search for the most similar image-text pairs in the candidate set to obtain the rich semantic image-text anchors related to the downstream task for regularizing the finetuning process.
  • Figure 4: The ID and domain shift performance of our ARF compared with several baselines through linear interpolation of the finetuned model weights with the original model weights following Wise-FT Wise-FT. The performance curves of ARF surpass (positioned in the upper right) those of the baselines on ImageNet, resulting in improved ID and domain shift accuracy.
  • Figure 5: Visualization examples of captions generated by a pretrained captioner (e.g., BLIP2 BLIP2) and retrieved image-text pairs relative to the downstream task from the CC3M dataset CC3M. The downstream task images and generated captions serve as text-compensated anchors for regularization. The retrieved image-text pairs function as auxiliary anchors for maintaining the feature space.