Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data

Badhan Kumar Das, Gengyan Zhao, Boris Mailhe, Thomas J. Re, Dorin Comaniciu, Eli Gibson, Andreas Maier

TL;DR

3D multi-plane vision transformer (MP-ViT) is proposed for hemorrhage classification with varying orientation data, which employs two separate transformer encoders for axial and sagittal contrasts, using cross-attention to integrate information across orientations.

Abstract

Identifying brain hemorrhages from magnetic resonance imaging (MRI) is a critical task for healthcare professionals. The diverse nature of MRI acquisitions with varying contrasts and orientation introduce complexity in identifying hemorrhage using neural networks. For acquisitions with varying orientations, traditional methods often involve resampling images to a fixed plane, which can lead to information loss. To address this, we propose a 3D multi-plane vision transformer (MP-ViT) for hemorrhage classification with varying orientation data. It employs two separate transformer encoders for axial and sagittal contrasts, using cross-attention to integrate information across orientations. MP-ViT also includes a modality indication vector to provide missing contrast information to the model. The effectiveness of the proposed model is demonstrated with extensive experiments on real world clinical dataset consists of 10,084 training, 1,289 validation and 1,496 test subjects. MP-ViT achieved substantial improvement in area under the curve (AUC), outperforming the vision transformer (ViT) by 5.5% and CNN-based architectures by 1.8%. These results highlight the potential of MP-ViT in improving performance for hemorrhage detection when different orientation contrasts are needed.

Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data

TL;DR

3D multi-plane vision transformer (MP-ViT) is proposed for hemorrhage classification with varying orientation data, which employs two separate transformer encoders for axial and sagittal contrasts, using cross-attention to integrate information across orientations.

Abstract

Identifying brain hemorrhages from magnetic resonance imaging (MRI) is a critical task for healthcare professionals. The diverse nature of MRI acquisitions with varying contrasts and orientation introduce complexity in identifying hemorrhage using neural networks. For acquisitions with varying orientations, traditional methods often involve resampling images to a fixed plane, which can lead to information loss. To address this, we propose a 3D multi-plane vision transformer (MP-ViT) for hemorrhage classification with varying orientation data. It employs two separate transformer encoders for axial and sagittal contrasts, using cross-attention to integrate information across orientations. MP-ViT also includes a modality indication vector to provide missing contrast information to the model. The effectiveness of the proposed model is demonstrated with extensive experiments on real world clinical dataset consists of 10,084 training, 1,289 validation and 1,496 test subjects. MP-ViT achieved substantial improvement in area under the curve (AUC), outperforming the vision transformer (ViT) by 5.5% and CNN-based architectures by 1.8%. These results highlight the potential of MP-ViT in improving performance for hemorrhage detection when different orientation contrasts are needed.
Paper Structure (17 sections, 7 equations, 3 figures, 4 tables)

This paper contains 17 sections, 7 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Results
  4. Discussion
  5. Conclusion
  6. Data availability
  7. Acknowledgements
  8. Author contributions statement
  9. Funding
  10. Competing interests
  11. Additional information
  12. Code availability

Figures (3)

  • Figure 1: Overview of multi-plane vision transformer with axial and sagittal transformer encoder
  • Figure 2: Overview of the cross attention fusion used in MP-ViT. In this block, the CLS token of the axial encoder input performs attention with the other tokens of the sagittal encoder. Similary, this process is done also with the CLS token of sagittal encoder input with the other tokens of the axial encoder. Here Wq, Wk and Wv are learnable matrices to create query, key and values for attention.
  • Figure 3: Receiver Operating Characteristic (ROC) Curve for different methods for hemorrhage classification