Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review

Iryna Hartsock, Ghulam Rasool

TL;DR

Medical vision-language models (VLMs) aim to fuse visual and textual data to support radiology report generation and medical visual question answering. The paper surveys architecture choices (single- vs dual-stream, encoder vs encoder–decoder), pre-training tasks (contrastive learning, MLM, MIM, ITM), and fine-tuning strategies (SFT, RLHF, IFT, PEFT) in the medical domain, and catalogs 17 public datasets and 15 models used for RG and VQA. It synthesizes evaluation metrics such as BLEU, ROUGE, METEOR, RadGraph F1, and VQA-specific measures, highlighting the importance of clinical relevance and human assessment. The review identifies challenges including data scarcity, patient privacy, hallucinations, and catastrophic forgetting, and advocates future directions like retrieval-augmented pre-training, federated learning, and robust clinical validation to enable safe, effective deployment of medical VLMs.

Abstract

Medical vision-language models (VLMs) combine computer vision (CV) and natural language processing (NLP) to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on models designed for medical report generation and visual question answering (VQA). We provide background on NLP and CV, explaining how techniques from both fields are integrated into VLMs to enable learning from multimodal data. Key areas we address include the exploration of medical vision-language datasets, in-depth analyses of architectures and pre-training strategies employed in recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and VQA. We also highlight current challenges and propose future directions, including enhancing clinical validity and addressing patient privacy concerns. Overall, our review summarizes recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.

Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review

TL;DR

Medical vision-language models (VLMs) aim to fuse visual and textual data to support radiology report generation and medical visual question answering. The paper surveys architecture choices (single- vs dual-stream, encoder vs encoder–decoder), pre-training tasks (contrastive learning, MLM, MIM, ITM), and fine-tuning strategies (SFT, RLHF, IFT, PEFT) in the medical domain, and catalogs 17 public datasets and 15 models used for RG and VQA. It synthesizes evaluation metrics such as BLEU, ROUGE, METEOR, RadGraph F1, and VQA-specific measures, highlighting the importance of clinical relevance and human assessment. The review identifies challenges including data scarcity, patient privacy, hallucinations, and catastrophic forgetting, and advocates future directions like retrieval-augmented pre-training, federated learning, and robust clinical validation to enable safe, effective deployment of medical VLMs.

Abstract

Medical vision-language models (VLMs) combine computer vision (CV) and natural language processing (NLP) to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on models designed for medical report generation and visual question answering (VQA). We provide background on NLP and CV, explaining how techniques from both fields are integrated into VLMs to enable learning from multimodal data. Key areas we address include the exploration of medical vision-language datasets, in-depth analyses of architectures and pre-training strategies employed in recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and VQA. We also highlight current challenges and propose future directions, including enhancing clinical validity and addressing patient privacy concerns. Overall, our review summarizes recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.
Paper Structure (94 sections, 9 equations, 1 figure, 3 tables)

This paper contains 94 sections, 9 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Machine Learning (ML) - A Brief Review
  3. Neural Networks (NNs)
  4. Natural Language Processing (NLP)
  5. Tokenization
  6. Token Embeddings
  7. Recurrent Neural Networks (RNNs)
  8. Transformers
  9. Computer Vision (CV)
  10. Convolutional Neural Networks (CNNs)
  11. Vision Transformers (ViTs)
  12. Vision-Language Models (VLMs)
  13. Model Architecture
  14. Single- vs. Dual-Stream VLMs
  15. Single-Stream Models
  16. ...and 79 more sections

Figures (1)

  • Figure 1: Organization of the review paper.