Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training

Che Liu, Cheng Ouyang, Sibo Cheng, Anand Shah, Wenjia Bai, Rossella Arcucci

TL;DR

A new VLP framework, named G2D, is proposed that achieves significantly improved granularity and more accurate grounding for the learned features, compared to existing medical VLP approaches, and achieves superior performance across 6 medical imaging tasks and 25 diseases.

Abstract

Recently, medical vision-language pre-training (VLP) has reached substantial progress to learn global visual representation from medical images and their paired radiology reports. However, medical imaging tasks in real world usually require finer granularity in visual features. These tasks include visual localization tasks (e.g., semantic segmentation, object detection) and visual grounding task. Yet, current medical VLP methods face challenges in learning these fine-grained features, as they primarily focus on brute-force alignment between image patches and individual text tokens for local visual feature learning, which is suboptimal for downstream dense prediction tasks. In this work, we propose a new VLP framework, named \textbf{G}lobal to \textbf{D}ense level representation learning (G2D) that achieves significantly improved granularity and more accurate grounding for the learned features, compared to existing medical VLP approaches. In particular, G2D learns dense and semantically-grounded image representations via a pseudo segmentation task parallel with the global vision-language alignment. Notably, generating pseudo segmentation targets does not incur extra trainable parameters: they are obtained on the fly during VLP with a parameter-free processor. G2D achieves superior performance across 6 medical imaging tasks and 25 diseases, particularly in semantic segmentation, which necessitates fine-grained, semantically-grounded image features. In this task, G2D surpasses peer models even when fine-tuned with just 1\% of the training data, compared to the 100\% used by these models. The code can be found in https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main.

G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training

TL;DR

A new VLP framework, named G2D, is proposed that achieves significantly improved granularity and more accurate grounding for the learned features, compared to existing medical VLP approaches, and achieves superior performance across 6 medical imaging tasks and 25 diseases.

Abstract

Recently, medical vision-language pre-training (VLP) has reached substantial progress to learn global visual representation from medical images and their paired radiology reports. However, medical imaging tasks in real world usually require finer granularity in visual features. These tasks include visual localization tasks (e.g., semantic segmentation, object detection) and visual grounding task. Yet, current medical VLP methods face challenges in learning these fine-grained features, as they primarily focus on brute-force alignment between image patches and individual text tokens for local visual feature learning, which is suboptimal for downstream dense prediction tasks. In this work, we propose a new VLP framework, named \textbf{G}lobal to \textbf{D}ense level representation learning (G2D) that achieves significantly improved granularity and more accurate grounding for the learned features, compared to existing medical VLP approaches. In particular, G2D learns dense and semantically-grounded image representations via a pseudo segmentation task parallel with the global vision-language alignment. Notably, generating pseudo segmentation targets does not incur extra trainable parameters: they are obtained on the fly during VLP with a parameter-free processor. G2D achieves superior performance across 6 medical imaging tasks and 25 diseases, particularly in semantic segmentation, which necessitates fine-grained, semantically-grounded image features. In this task, G2D surpasses peer models even when fine-tuned with just 1\% of the training data, compared to the 100\% used by these models. The code can be found in https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main.
Paper Structure (24 sections, 6 equations, 4 figures, 8 tables)

This paper contains 24 sections, 6 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Vision-Language Contrastive Learning
  5. Pseudo Segmentation Mask Construction
  6. Dense Visual Representation Learning through Pseudo Segmentation in VLP
  7. Experiments and Analysis
  8. Downstream Task Datasets and Configurations
  9. Performance on Visual Localisation Tasks
  10. Performance on Vision-Language Understanding
  11. Performance on Visual Recognition Tasks
  12. Ablation Studies
  13. Conclusion
  14. Appendix / supplemental material
  15. Limitations and Future Work
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Comparing existing medical VLP methods with G2D: a) Alignment-based approaches lack dense (pixel-level) feature learning. b) Reconstruction-based approaches do not align with text, resulting in a deficiency in discriminative and clinically relevant visual features. c) The framework of G2D (proposed) learns dense, clinically relevant, text-aligned visual features through derived pseudo masks and image-text alignment. We use red text to highlight the deficiencies of existing methods and blue text to emphasize our advantages.
  • Figure 2: Left: Framework of G2D. Right: Pipeline for pseudo mask construction. We visualize the constructed pseudo mask and corresponding sentence in the radiology report in Sec \ref{['sec: pseudo mask vis']}.
  • Figure 3: An exemplar pair of X-ray image and associated clinical report from the MIMIC-CXR dataset johnson2019mimic.
  • Figure 4: Pseudo Mask Visualization. Left: Aggregated attention map. Middle: Constructed pseudo mask for the pseudo segmentation task. Red and blue arrows point to areas related to specific text descriptions. Right: Corresponding radiology report. Red and blue text emphasize regions represented in the pseudo mask.