Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Medical Vision-Language Pre-Training for Brain Abnormalities

Masoud Monajatipoor, Zi-Yi Dou, Aichi Chien, Nanyun Peng, Kai-Wei Chang

TL;DR

The paper tackles the scarcity of domain-specific vision-language resources for medical tasks by proposing an automatic pipeline to harvest brain-related image-text data from PubMed and related sources. It introduces a subfigure-subcaption alignment mechanism to handle medical publications' complex figure layouts and trains a BLIP-based VL model on both raw and processed data. Quantitative image-text retrieval and qualitative attention analyses demonstrate data-efficient improvements in medical multimodal understanding, with a notable emphasis on precise localization of clinical terms and abnormalities. The work provides a data and code release to enable domain-specific VL pre-training for brain diseases and potentially other medical domains, enhancing capabilities for diagnosis and information extraction in multimodal clinical AI.

Abstract

Vision-language models have become increasingly powerful for tasks that require an understanding of both visual and linguistic elements, bridging the gap between these modalities. In the context of multimodal clinical AI, there is a growing need for models that possess domain-specific knowledge, as existing models often lack the expertise required for medical applications. In this paper, we take brain abnormalities as an example to demonstrate how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset from case reports and published journals and subsequently constructing a high-performance vision-language model tailored to specific medical tasks. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain. We evaluated the resulting model with quantitative and qualitative intrinsic evaluations. The resulting dataset and our code can be found here https://github.com/masoud-monajati/MedVL_pretraining_pipeline

Medical Vision-Language Pre-Training for Brain Abnormalities

TL;DR

The paper tackles the scarcity of domain-specific vision-language resources for medical tasks by proposing an automatic pipeline to harvest brain-related image-text data from PubMed and related sources. It introduces a subfigure-subcaption alignment mechanism to handle medical publications' complex figure layouts and trains a BLIP-based VL model on both raw and processed data. Quantitative image-text retrieval and qualitative attention analyses demonstrate data-efficient improvements in medical multimodal understanding, with a notable emphasis on precise localization of clinical terms and abnormalities. The work provides a data and code release to enable domain-specific VL pre-training for brain diseases and potentially other medical domains, enhancing capabilities for diagnosis and information extraction in multimodal clinical AI.

Abstract

Vision-language models have become increasingly powerful for tasks that require an understanding of both visual and linguistic elements, bridging the gap between these modalities. In the context of multimodal clinical AI, there is a growing need for models that possess domain-specific knowledge, as existing models often lack the expertise required for medical applications. In this paper, we take brain abnormalities as an example to demonstrate how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset from case reports and published journals and subsequently constructing a high-performance vision-language model tailored to specific medical tasks. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain. We evaluated the resulting model with quantitative and qualitative intrinsic evaluations. The resulting dataset and our code can be found here https://github.com/masoud-monajati/MedVL_pretraining_pipeline
Paper Structure (15 sections, 5 figures, 1 table)

This paper contains 15 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Brain Image-Caption for VL Pre-Training
  3. Data collection and processing
  4. Image-caption pairs.
  5. Fine-grained alignment.
  6. Pre-Training and Evaluation
  7. Results
  8. Quantitaive evaluation
  9. Qualitative analysis
  10. Related Work
  11. Vision-Language Learning.
  12. Medical-Domain VL Learning.
  13. Conclusion
  14. Aknowledgment
  15. References

Figures (5)

  • Figure 1: Our main pipeline for matching subfigures/subcaptions. The object detector outputs subfigures while the caption parser parses the caption into subcaptions simultaneously. Then, the module called 'matching' provides us the subfigure/subcaption pairs for the pre-training
  • Figure 2: An example data from the dataset where each subfigure comes from a different source and Providing the model with the entire figure and its accompanying caption could potentially result in misdirection, causing it to emphasize image areas that are less relevant to both the caption and subcaptions.
  • Figure 3: The plot visually demonstrates the dynamics of image-text retrieval performance across varying proportions of pre-training data. Our model results reveal a notable increase in learning capacity as more data becomes available, while the baseline exhibits characteristics that suggest a form of saturation.
  • Figure 4: The visualization of the attention of the "cerebral artery" on the corresponding image. It is evident that ours highlighted the abnormal region more specifically.
  • Figure 5: The visualization of the attention of the "aneurysm" on the corresponding image. It is evident that ours highlighted the abnormal region more specifically.