Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning

Yang Bai, Yang Zhou, Jun Zhou, Rick Siow Mong Goh, Daniel Shu Wei Ting, Yong Liu

TL;DR

This work tackles the gap between pre-trained vision-language models and task-specific performance by introducing VITask, a framework that tightly integrates task-specific models with VLMs through Exemplar Prompting (EP), Response Distribution Alignment (RDA), and Contrastive Response Tuning (CRT). By decoupling image representation learning from instruction tuning and employing a two-stage training pipeline, VITask leverages task-specific features without altering the VLM’s vision encoder, enabling strong task-focused performance while preserving generalist capabilities. Empirically, it achieves state-of-the-art or competitive results across 12 medical diagnostic datasets spanning 9 imaging modalities, and demonstrates robustness to incomplete instructions and flexibility to incorporate new TSMs with minimal retraining. The approach offers a practical, scalable pathway to adapt VLMs to new domains while maintaining broad instruction-following behavior, with potential applicability beyond medicine.

Abstract

Large vision language models (VLMs) combine large language models with vision encoders, demonstrating promise across various tasks. However, they often underperform in task-specific applications due to domain gaps between pre-training and fine-tuning. We introduce VITask, a novel framework that enhances task-specific adaptability of VLMs by integrating task-specific models (TSMs). VITask employs three key strategies: exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT) to improve the task-specific performance of VLMs by adjusting their response distributions. EP allows TSM features to guide VLMs, while RDA enables VLMs to adapt without TSMs during inference by learning from exemplar-prompted models. CRT further optimizes the ranking of correct image-response pairs, thereby reducing the risk of generating undesired responses. Experiments on 12 medical diagnosis datasets across 9 imaging modalities show that VITask outperforms both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to integrate complementary features from both models effectively. Additionally, VITask offers practical advantages such as flexible TSM integration and robustness to incomplete instructions, making it a versatile and efficient solution for task-specific VLM tuning. Our code are available at https://github.com/baiyang4/VITask.

From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning

TL;DR

This work tackles the gap between pre-trained vision-language models and task-specific performance by introducing VITask, a framework that tightly integrates task-specific models with VLMs through Exemplar Prompting (EP), Response Distribution Alignment (RDA), and Contrastive Response Tuning (CRT). By decoupling image representation learning from instruction tuning and employing a two-stage training pipeline, VITask leverages task-specific features without altering the VLM’s vision encoder, enabling strong task-focused performance while preserving generalist capabilities. Empirically, it achieves state-of-the-art or competitive results across 12 medical diagnostic datasets spanning 9 imaging modalities, and demonstrates robustness to incomplete instructions and flexibility to incorporate new TSMs with minimal retraining. The approach offers a practical, scalable pathway to adapt VLMs to new domains while maintaining broad instruction-following behavior, with potential applicability beyond medicine.

Abstract

Large vision language models (VLMs) combine large language models with vision encoders, demonstrating promise across various tasks. However, they often underperform in task-specific applications due to domain gaps between pre-training and fine-tuning. We introduce VITask, a novel framework that enhances task-specific adaptability of VLMs by integrating task-specific models (TSMs). VITask employs three key strategies: exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT) to improve the task-specific performance of VLMs by adjusting their response distributions. EP allows TSM features to guide VLMs, while RDA enables VLMs to adapt without TSMs during inference by learning from exemplar-prompted models. CRT further optimizes the ranking of correct image-response pairs, thereby reducing the risk of generating undesired responses. Experiments on 12 medical diagnosis datasets across 9 imaging modalities show that VITask outperforms both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to integrate complementary features from both models effectively. Additionally, VITask offers practical advantages such as flexible TSM integration and robustness to incomplete instructions, making it a versatile and efficient solution for task-specific VLM tuning. Our code are available at https://github.com/baiyang4/VITask.
Paper Structure (32 sections, 7 equations, 5 figures, 3 tables)

This paper contains 32 sections, 7 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision Language Models.
  4. Visual Instruction Tuning.
  5. Integrating VLMs and TSMs.
  6. Instruction-Tuned VLMs vs. Task-Specific Models
  7. Setting.
  8. Instruction Formatting.
  9. Model Training.
  10. Observations.
  11. Task-Specific Visual Instruction Tuning
  12. Exemplar Prompting.
  13. Implementation and Analysis.
  14. Response Distribution Alignment.
  15. Implementation and Analysis.
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Overview of the proposed VITask framework. (a) Traditional visual instruction tuning. (b) Exemplar Prompting (EP) enhances VLM’s image representations using TSM features without modifying pre-trained features. (c) Response Distribution Alignment (RDA) aligns EP and non-EP responses to capture task-specific information. (d) Contrastive Response Tuning (CRT) leverages negative samples to improve the VLM’s response ranking capability by maximizing the margin between correct and incorrect image-response pairs.
  • Figure 2: Illustration of the performance discrepancy between TSM and VLMs.
  • Figure 3: Illustration on how CRT improves the visual response ranking capability for VLMs.
  • Figure 4: Robustness to incomplete instructions.
  • Figure 5: Performance of VITask in adapting to different tasks.