Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Supervised Fine-tuning in turn Improves Visual Foundation Models

Xiaohu Jiang, Yixiao Ge, Yuying Ge, Dachuan Shi, Chun Yuan, Ying Shan

TL;DR

This work proposes ViSFT, a two-stage supervised fine-tuning approach to enhance vision foundation models after image-text pretraining. By first training in-domain task heads with a frozen backbone and then updating the backbone with LoRA while keeping heads fixed, ViSFT transfers fine-grained knowledge into a lightweight parameter budget. Across OCR, GOI, zero-shot and few-shot classification, image-text retrieval, and VQA, ViSFT yields consistent out-of-domain gains on EVA-ViT backbones exceeding $4.4$B parameters, using a modest computational setup. The study demonstrates the value of targeted, stage-wise fine-tuning for unlocking hidden, fine-grained information in vision transformers to improve generalization without large-scale region-level data.

Abstract

Image-text training like CLIP has dominated the pretraining of vision foundation models in recent years. Subsequent efforts have been made to introduce region-level visual learning into CLIP's pretraining but face scalability challenges due to the lack of large-scale region-level datasets. Drawing inspiration from supervised fine-tuning (SFT) in natural language processing such as instruction tuning, we explore the potential of fine-grained SFT in enhancing the generation of vision foundation models after their pretraining. Thus a two-stage method ViSFT (Vision SFT) is proposed to unleash the fine-grained knowledge of vision foundation models. In ViSFT, the vision foundation model is enhanced by performing visual joint learning on some in-domain tasks and then tested on out-of-domain benchmarks. With updating using ViSFT on 8 V100 GPUs in less than 2 days, a vision transformer with over 4.4B parameters shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios.

Supervised Fine-tuning in turn Improves Visual Foundation Models

TL;DR

This work proposes ViSFT, a two-stage supervised fine-tuning approach to enhance vision foundation models after image-text pretraining. By first training in-domain task heads with a frozen backbone and then updating the backbone with LoRA while keeping heads fixed, ViSFT transfers fine-grained knowledge into a lightweight parameter budget. Across OCR, GOI, zero-shot and few-shot classification, image-text retrieval, and VQA, ViSFT yields consistent out-of-domain gains on EVA-ViT backbones exceeding B parameters, using a modest computational setup. The study demonstrates the value of targeted, stage-wise fine-tuning for unlocking hidden, fine-grained information in vision transformers to improve generalization without large-scale region-level data.

Abstract

Image-text training like CLIP has dominated the pretraining of vision foundation models in recent years. Subsequent efforts have been made to introduce region-level visual learning into CLIP's pretraining but face scalability challenges due to the lack of large-scale region-level datasets. Drawing inspiration from supervised fine-tuning (SFT) in natural language processing such as instruction tuning, we explore the potential of fine-grained SFT in enhancing the generation of vision foundation models after their pretraining. Thus a two-stage method ViSFT (Vision SFT) is proposed to unleash the fine-grained knowledge of vision foundation models. In ViSFT, the vision foundation model is enhanced by performing visual joint learning on some in-domain tasks and then tested on out-of-domain benchmarks. With updating using ViSFT on 8 V100 GPUs in less than 2 days, a vision transformer with over 4.4B parameters shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios.
Paper Structure (17 sections, 3 equations, 3 figures, 10 tables, 3 algorithms)

This paper contains 17 sections, 3 equations, 3 figures, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Pretraining of Vision Foundation Model
  4. Visual-Linguistic Instruction Tuning
  5. Multi-Task Training
  6. Method
  7. Tasks and Datasets
  8. Model Details
  9. Main Experiments
  10. Evaluation Benchmarks
  11. Implementation Details
  12. Main Results
  13. Ablation Studies
  14. Visualization
  15. Conclusion
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Drawing inspiration from the training paradigm in NLP, we perform ViSFT on vision foundation models after their pretraining and subsequently evaluate them on out-of-domain tasks.
  • Figure 2: An overview of our proposed method is as follows: (a) First, a vision foundation model is pretrained such as CLIP-ViT. (b) Next, we execute ViSFT to update the LoRA weights and retain the fine-grained information through joint learning of in-domain tasks. (c) Finally, in conjunction with the updated LoRA weights, evaluations on multiple out-of-domain tasks exhibit considerable enhancement. "OCR" refers to the optical character recognition task, while "GOI" denotes the grounded object identification task.
  • Figure 3: Visualization of [CLS] token's attention distribution. Experiments are conducted on the last layer of EVA-ViT-G. Attended image patches are highlighted.