PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging

TL;DR

PeFoMed introduces a PEFT-based approach for adapting multimodal large language models to medical imaging tasks, achieving strong Med-VQA and MRG performance with a minimal trainable footprint by freezing the vision encoder and LLM and updating only a projection layer and LoRA adapters. The method uses a two-stage training regime—image-captioning followed by downstream Med-VQA/MRG fine-tuning—and task-specific multimodal prompts, evaluated with both human and GPT-4 semantic similarity against traditional lexical metrics. Results show GPT-4 semantic similarity aligns closely with human judgments and is more stable than lexical metrics, while fine-tuned models can outperform GPT-4v in medical imaging tasks. The work demonstrates the practicality of PEFT for medical multimodal tasks and contributes a robust evaluation framework, including a public code release.

Abstract

Multimodal large language models (MLLMs) represent an evolutionary expansion in the capabilities of traditional large language models, enabling them to tackle challenges that surpass the scope of purely text-based applications. It leverages the knowledge previously encoded within these language models, thereby enhancing their applicability and functionality in the reign of multimodal contexts. Recent works investigate the adaptation of MLLMs as a universal solution to address medical multi-modal problems as a generative task. In this paper, we propose a parameter efficient framework for fine-tuning MLLMs, specifically validated on medical visual question answering (Med-VQA) and medical report generation (MRG) tasks, using public benchmark datasets. We also introduce an evaluation metric using the 5-point Likert scale and its weighted average value to measure the quality of the generated reports for MRG tasks, where the scale ratings are labelled by both humans manually and the GPT-4 model. We further assess the consistency of performance metrics across traditional measures, GPT-4, and human ratings for both VQA and MRG tasks. The results indicate that semantic similarity assessments using GPT-4 align closely with human annotators and provide greater stability, yet they reveal a discrepancy when compared to conventional lexical similarity measurements. This questions the reliability of lexical similarity metrics for evaluating the performance of generative models in Med-VQA and report generation tasks. Besides, our fine-tuned model significantly outperforms GPT-4v. This indicates that without additional fine-tuning, multi-modal models like GPT-4v do not perform effectively on medical imaging tasks. The code will be available here: https://github.com/jinlHe/PeFoMed.

Figures (8)

• Figure 1: The architecture of the model.
• Figure 2: 5-point Likert scale for semantic similarity, used to evaluate medical reports.
• Figure 3: The accuracy of different question types on the VQA-RAD dataset by different evaluation methods.
• Figure 4: Accuracy of different evaluation methods for different types of questions on the PathVQA dataset.
• Figure 5: The accuracy of different evaluation methods for different question types on the Slake dataset.