Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLMs' Tuning Methods Work in Medical Multimodal Domain?

Jiawei Chen, Yue Jiang, Dingkang Yang, Mingcheng Li, Jinjie Wei, Ziyun Qian, Lihua Zhang

TL;DR

This paper tackles the challenge of transferring parameter-efficient tuning (PEFT) methods from large language models to medical multimodal vision-language models to reduce training costs and address data privacy concerns. It introduces MILE, a modular fine-tuning framework built on a small-scale medical VLP baseline (MISS) and augmented with four PEFT modules (LoRA, IA3, Prefix, P-Tuning v2) plus an instruction-format data pipeline. Through extensive experiments on radiographic benchmarks (Slake and VQA-RAD), the study finds that updating the visual encoder and the JTM encoder is crucial, with LoRA and Prefix-tuning often delivering competitive performance compared with global fine-tuning while significantly reducing training cost. The results provide nuanced guidance on when data-level instruction-tuning helps or hinders performance for basic VLPs, offering practical strategies for cost-conscious development of medical multimodal models and publishing datasets and code to enable reproducibility and broader adoption.

Abstract

While Large Language Models (LLMs) excel in world knowledge understanding, adapting them to specific subfields requires precise adjustments. Due to the model's vast scale, traditional global fine-tuning methods for large models can be computationally expensive and impact generalization. To address this challenge, a range of innovative Parameters-Efficient Fine-Tuning (PEFT) methods have emerged and achieved remarkable success in both LLMs and Large Vision-Language Models (LVLMs). In the medical domain, fine-tuning a medical Vision-Language Pretrained (VLP) model is essential for adapting it to specific tasks. Can the fine-tuning methods for large models be transferred to the medical field to enhance transfer learning efficiency? In this paper, we delve into the fine-tuning methods of LLMs and conduct extensive experiments to investigate the impact of fine-tuning methods for large models on the existing multimodal model in the medical domain from the training data level and the model structure level. We show the different impacts of fine-tuning methods for large models on medical VLMs and develop the most efficient ways to fine-tune medical VLP models. We hope this research can guide medical domain researchers in optimizing VLMs' training costs, fostering the broader application of VLMs in healthcare fields. The code and dataset have been released at https://github.com/TIMMY-CHAN/MILE.

Can LLMs' Tuning Methods Work in Medical Multimodal Domain?

TL;DR

This paper tackles the challenge of transferring parameter-efficient tuning (PEFT) methods from large language models to medical multimodal vision-language models to reduce training costs and address data privacy concerns. It introduces MILE, a modular fine-tuning framework built on a small-scale medical VLP baseline (MISS) and augmented with four PEFT modules (LoRA, IA3, Prefix, P-Tuning v2) plus an instruction-format data pipeline. Through extensive experiments on radiographic benchmarks (Slake and VQA-RAD), the study finds that updating the visual encoder and the JTM encoder is crucial, with LoRA and Prefix-tuning often delivering competitive performance compared with global fine-tuning while significantly reducing training cost. The results provide nuanced guidance on when data-level instruction-tuning helps or hinders performance for basic VLPs, offering practical strategies for cost-conscious development of medical multimodal models and publishing datasets and code to enable reproducibility and broader adoption.

Abstract

While Large Language Models (LLMs) excel in world knowledge understanding, adapting them to specific subfields requires precise adjustments. Due to the model's vast scale, traditional global fine-tuning methods for large models can be computationally expensive and impact generalization. To address this challenge, a range of innovative Parameters-Efficient Fine-Tuning (PEFT) methods have emerged and achieved remarkable success in both LLMs and Large Vision-Language Models (LVLMs). In the medical domain, fine-tuning a medical Vision-Language Pretrained (VLP) model is essential for adapting it to specific tasks. Can the fine-tuning methods for large models be transferred to the medical field to enhance transfer learning efficiency? In this paper, we delve into the fine-tuning methods of LLMs and conduct extensive experiments to investigate the impact of fine-tuning methods for large models on the existing multimodal model in the medical domain from the training data level and the model structure level. We show the different impacts of fine-tuning methods for large models on medical VLMs and develop the most efficient ways to fine-tune medical VLP models. We hope this research can guide medical domain researchers in optimizing VLMs' training costs, fostering the broader application of VLMs in healthcare fields. The code and dataset have been released at https://github.com/TIMMY-CHAN/MILE.
Paper Structure (14 sections, 5 figures, 8 tables)

This paper contains 14 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Architecture of MILE
  5. Instruction-format Data Generation
  6. Training
  7. Experiment Results and Analysis
  8. Performance Differences Among Different PEFT Methods
  9. The Impact of Data on Training Effectiveness
  10. Is Instruction-Tuning Ineffective for Basic VLMs?
  11. Conclusion
  12. Acknowledgement
  13. Templates of instruction-format data
  14. Details of Experiments

Figures (5)

  • Figure 1: Architecture of MILE with different PEFT modules (a) and PEFT details in a Transformer layer (b).
  • Figure 2: Results of MILE-Prefix.
  • Figure 3: Loss of MILE variants under the training with origin data (a), instruction-format data (b) and an instruction-tuning MILE after fine-tuning by origin data (c).
  • Figure 4: Instruction-format Data Templates.
  • Figure 5: Instruction-format Data Examples.