Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tumor Location-weighted MRI-Report Contrastive Learning: A Framework for Improving the Explainability of Pediatric Brain Tumor Diagnosis

Sara Ketabi, Matthias W. Wagner, Cynthia Hawkins, Uri Tabori, Birgit Betina Ertl-Wagner, Farzad Khalvati

TL;DR

This work trains a multimodal CL architecture on 3D brain MRI scans and radiology reports to learn informative MRI representations and integrates tumor location, salient to several brain tumor analysis tasks, into this framework to improve its generalizability.

Abstract

Despite the promising performance of convolutional neural networks (CNNs) in brain tumor diagnosis from magnetic resonance imaging (MRI), their integration into the clinical workflow has been limited. That is mainly due to the fact that the features contributing to a model's prediction are unclear to radiologists and hence, clinically irrelevant, i.e., lack of explainability. As the invaluable sources of radiologists' knowledge and expertise, radiology reports can be integrated with MRI in a contrastive learning (CL) framework, enabling learning from image-report associations, to improve CNN explainability. In this work, we train a multimodal CL architecture on 3D brain MRI scans and radiology reports to learn informative MRI representations. Furthermore, we integrate tumor location, salient to several brain tumor analysis tasks, into this framework to improve its generalizability. We then apply the learnt image representations to improve explainability and performance of genetic marker classification of pediatric Low-grade Glioma, the most prevalent brain tumor in children, as a downstream task. Our results indicate a Dice score of 31.1% between the model's attention maps and manual tumor segmentation (as an explainability measure) with test classification performance of 87.7%, significantly outperforming the baselines. These enhancements can build trust in our model among radiologists, facilitating its integration into clinical practices for more efficient tumor diagnosis.

Tumor Location-weighted MRI-Report Contrastive Learning: A Framework for Improving the Explainability of Pediatric Brain Tumor Diagnosis

TL;DR

This work trains a multimodal CL architecture on 3D brain MRI scans and radiology reports to learn informative MRI representations and integrates tumor location, salient to several brain tumor analysis tasks, into this framework to improve its generalizability.

Abstract

Despite the promising performance of convolutional neural networks (CNNs) in brain tumor diagnosis from magnetic resonance imaging (MRI), their integration into the clinical workflow has been limited. That is mainly due to the fact that the features contributing to a model's prediction are unclear to radiologists and hence, clinically irrelevant, i.e., lack of explainability. As the invaluable sources of radiologists' knowledge and expertise, radiology reports can be integrated with MRI in a contrastive learning (CL) framework, enabling learning from image-report associations, to improve CNN explainability. In this work, we train a multimodal CL architecture on 3D brain MRI scans and radiology reports to learn informative MRI representations. Furthermore, we integrate tumor location, salient to several brain tumor analysis tasks, into this framework to improve its generalizability. We then apply the learnt image representations to improve explainability and performance of genetic marker classification of pediatric Low-grade Glioma, the most prevalent brain tumor in children, as a downstream task. Our results indicate a Dice score of 31.1% between the model's attention maps and manual tumor segmentation (as an explainability measure) with test classification performance of 87.7%, significantly outperforming the baselines. These enhancements can build trust in our model among radiologists, facilitating its integration into clinical practices for more efficient tumor diagnosis.

Paper Structure

This paper contains 30 sections, 7 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Results
  4. Discussion
  5. Conclusion
  6. Data Availability Statement
  7. Figure Legends
  8. Acknowledgements
  9. Author contributions statement
  10. Additional information

Figures (3)

  • Figure 1: The Proposed MRI-Report Contrastive Learning Framework
  • Figure 2: The architecture of the downstream pLGG genetic marker classification model
  • Figure 3: From left to right: MRI slice, manual tumor segmentation mask slice, model attention map slice, and MRI with overlaid attention map, for a sample in Dataset 2, when 3D ResNet is initialized: (a) randomly, (b) with MedicalNet weights, and (c) with proposed CL weights