Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal self-supervised learning for lesion localization

Hao Yang, Hong-Yu Zhou, Cheng Li, Weijian Huang, Jiarun Liu, Yong Liang, Guangming Shi, Hairong Zheng, Qiegen Liu, Shanshan Wang

TL;DR

The paper addresses lesion localization in chest X-rays under weak supervision by leveraging multimodal data: images and radiology reports. It introduces a model that uses sentence-level semantic units for local alignment between text and image regions, coupled with global contrastive learning, and trains with a joint loss $L = L_g^{(v|t)}+L_g^{(t|v)}+L_l^{(v|t)}+L_l^{(t|v)}$. The approach, built on a ResNet-50 image encoder and BioClinicalBERT text encoder, demonstrates state-of-the-art zero-shot localization on RSNA Pneumonia, COVID Rural, and MS-CXR datasets, with robust performance on unseen diseases. The method’s ability to localize lesions using only descriptive sentences from reports suggests significant practical impact for rapid, annotation-light medical diagnostics and improved generalization to emerging diseases.

Abstract

Multimodal deep learning utilizing imaging and diagnostic reports has made impressive progress in the field of medical imaging diagnostics, demonstrating a particularly strong capability for auxiliary diagnosis in cases where sufficient annotation information is lacking. Nonetheless, localizing diseases accurately without detailed positional annotations remains a challenge. Although existing methods have attempted to utilize local information to achieve fine-grained semantic alignment, their capability in extracting the fine-grained semantics of the comprehensive context within reports is limited. To address this problem, a new method is introduced that takes full sentences from textual reports as the basic units for local semantic alignment. This approach combines chest X-ray images with their corresponding textual reports, performing contrastive learning at both global and local levels. The leading results obtained by this method on multiple datasets confirm its efficacy in the task of lesion localization.

Multimodal self-supervised learning for lesion localization

TL;DR

The paper addresses lesion localization in chest X-rays under weak supervision by leveraging multimodal data: images and radiology reports. It introduces a model that uses sentence-level semantic units for local alignment between text and image regions, coupled with global contrastive learning, and trains with a joint loss . The approach, built on a ResNet-50 image encoder and BioClinicalBERT text encoder, demonstrates state-of-the-art zero-shot localization on RSNA Pneumonia, COVID Rural, and MS-CXR datasets, with robust performance on unseen diseases. The method’s ability to localize lesions using only descriptive sentences from reports suggests significant practical impact for rapid, annotation-light medical diagnostics and improved generalization to emerging diseases.

Abstract

Multimodal deep learning utilizing imaging and diagnostic reports has made impressive progress in the field of medical imaging diagnostics, demonstrating a particularly strong capability for auxiliary diagnosis in cases where sufficient annotation information is lacking. Nonetheless, localizing diseases accurately without detailed positional annotations remains a challenge. Although existing methods have attempted to utilize local information to achieve fine-grained semantic alignment, their capability in extracting the fine-grained semantics of the comprehensive context within reports is limited. To address this problem, a new method is introduced that takes full sentences from textual reports as the basic units for local semantic alignment. This approach combines chest X-ray images with their corresponding textual reports, performing contrastive learning at both global and local levels. The leading results obtained by this method on multiple datasets confirm its efficacy in the task of lesion localization.
Paper Structure (12 sections, 8 equations, 3 figures, 3 tables)

This paper contains 12 sections, 8 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Image Encoder
  4. Text Encoder
  5. Global and Local contrastive learning
  6. EXPERIMENTS AND RESULTS
  7. Dataset
  8. Experimental details
  9. Results
  10. CONCLUSION
  11. Compliance with ethical standards
  12. Acknowledgments

Figures (3)

  • Figure 1: The description in the diagnostic report typically corresponds to the findings in the imaging.
  • Figure 2: The framework is based on global and local contrastive learning. Global contrastive learning uses global image features and text features. Local contrastive learning aligns sentence-level features with local image features.
  • Figure 3: Disease localization based on text descriptions. Subfigures a, b, and c display the results of disease localization for different pathological conditions.