Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VLLaVO: Mitigating Visual Gap through LLMs

Shuhao Chen, Yulong Zhang, Weisen Jiang, Jiangang Lu, Yu Zhang

TL;DR

This work proposes VLLaVO, combining Vision language models and Large Language models as Visual cross-dOmain learners, using vision-language models to convert images into detailed textual descriptions and finetuned on textual descriptions of the source/target domain generated by a designed instruction template.

Abstract

Recent advances achieved by deep learning models rely on the independent and identically distributed assumption, hindering their applications in real-world scenarios with domain shifts. To tackle this issue, cross-domain learning aims at extracting domain-invariant knowledge to reduce the domain shift between training and testing data. However, in visual cross-domain learning, traditional methods concentrate solely on the image modality, disregarding the potential benefits of incorporating the text modality. In this work, we propose VLLaVO, combining Vision language models and Large Language models as Visual cross-dOmain learners. VLLaVO uses vision-language models to convert images into detailed textual descriptions. A large language model is then finetuned on textual descriptions of the source/target domain generated by a designed instruction template. Extensive experimental results under domain generalization and unsupervised domain adaptation settings demonstrate the effectiveness of the proposed method.

VLLaVO: Mitigating Visual Gap through LLMs

TL;DR

This work proposes VLLaVO, combining Vision language models and Large Language models as Visual cross-dOmain learners, using vision-language models to convert images into detailed textual descriptions and finetuned on textual descriptions of the source/target domain generated by a designed instruction template.

Abstract

Recent advances achieved by deep learning models rely on the independent and identically distributed assumption, hindering their applications in real-world scenarios with domain shifts. To tackle this issue, cross-domain learning aims at extracting domain-invariant knowledge to reduce the domain shift between training and testing data. However, in visual cross-domain learning, traditional methods concentrate solely on the image modality, disregarding the potential benefits of incorporating the text modality. In this work, we propose VLLaVO, combining Vision language models and Large Language models as Visual cross-dOmain learners. VLLaVO uses vision-language models to convert images into detailed textual descriptions. A large language model is then finetuned on textual descriptions of the source/target domain generated by a designed instruction template. Extensive experimental results under domain generalization and unsupervised domain adaptation settings demonstrate the effectiveness of the proposed method.
Paper Structure (33 sections, 3 equations, 4 figures, 16 tables, 1 algorithm)

This paper contains 33 sections, 3 equations, 4 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overview of The Entire Framework
  5. Generating Textual Descriptions for Images
  6. Question Instruction
  7. Finetuning
  8. Finetuning for DG Tasks
  9. Finetuning for UDA Tasks
  10. Inference
  11. Experiments
  12. Experiments on DG Tasks
  13. Experiments on UDA Tasks
  14. Ablation Study
  15. Effects of Description Components
  16. ...and 18 more sections

Figures (4)

  • Figure 1: t-SNE visualizations of samples from the Art Painting domain (marked by "o") and the Cartoon domain (marked by "x") on the PACS dataset. Samples of the same category are depicted in the same color. Implementation details are provided in Section \ref{['sec:domain_invariant']}.
  • Figure 2: An illustration of the proposed VLLaVO framework for both UDA and DG. For DG, the source images are converted into the text modality through the VLMs. Then, the extracted descriptions are used to finetune an LLM. For UDA, we use the target domain samples with pseudo-labels and the source domain samples with ground truth labels to finetune the LLM.
  • Figure 3: Comparison between ZS-LLM and VLLaVO (finetuned on the PACS dataset).
  • Figure 4: t-SNE visualization of samples from the Real World domain (marked in red color) and the Art domain (marked in blue color) on the OfficeHome dataset.