Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification

Juan A. Olmos, Antoine Manzanera, Fabio Martínez

TL;DR

This work tackles csPCa classification by fusing bp-MRI with clinical reports through a geometric multimodal foundation model, MFM-Geom. It encodes imaging and text with BiomedCLIP and PubMedBERT, constructs a symmetric positive definite (SPD) descriptor S_0 = (1/d^2) M M^{\intercal} from the multimodal embeddings, and processes it with a BiMap-ReEig-LogEig-based geometric head for classification. A contrastive InfoNCE loss together with binary cross-entropy aligns image-text representations in a shared latent space, improving performance under limited data and yielding robust external generalization (AUC-PR ≈ 90.6 on PROSTATE158). The approach demonstrates enhanced detection of csPCa and especially intermediate-malignancy cases, with attention maps illustrating clinically meaningful cues such as PSAD, PV, and lesion location. This geometric fusion framework has potential to improve diagnostic accuracy and treatment decisions in clinical workflows where imaging is accompanied by rich clinical context.

Abstract

Prostate cancer (PCa) is one of the most common cancers in men worldwide. Bi-parametric MRI (bp-MRI) and clinical variables are crucial for PCa identification and improving treatment decisions. However, this process is subjective to expert interpretations. Furthermore, most existing computer-aided diagnosis methods focus on imaging-based models, overlooking the clinical context and suffering from data scarcity, limiting their ability to learn robust representations. We propose a geometric multimodal Foundation Model (FM), named MFM-Geom, that learns representations from bp-MRI and clinical reports, encoding visual findings and information from the context of clinical variables. In the representations classification head, the approach leverages symmetric positive definite (SPD) matrices and Riemannian deep learning to integrate imaging-text representations from a biomedical multimodal FM. Using 10% of the training data, MFM-Geom outperformed baseline class token embedding-based classification (+8.3%, AUC-PR of 90.67). Generalization on external dataset confirmed the robustness of fine-tuning biomedical FM, achieving an AUC-PR of 90.6.

A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification

TL;DR

This work tackles csPCa classification by fusing bp-MRI with clinical reports through a geometric multimodal foundation model, MFM-Geom. It encodes imaging and text with BiomedCLIP and PubMedBERT, constructs a symmetric positive definite (SPD) descriptor S_0 = (1/d^2) M M^{\intercal} from the multimodal embeddings, and processes it with a BiMap-ReEig-LogEig-based geometric head for classification. A contrastive InfoNCE loss together with binary cross-entropy aligns image-text representations in a shared latent space, improving performance under limited data and yielding robust external generalization (AUC-PR ≈ 90.6 on PROSTATE158). The approach demonstrates enhanced detection of csPCa and especially intermediate-malignancy cases, with attention maps illustrating clinically meaningful cues such as PSAD, PV, and lesion location. This geometric fusion framework has potential to improve diagnostic accuracy and treatment decisions in clinical workflows where imaging is accompanied by rich clinical context.

Abstract

Prostate cancer (PCa) is one of the most common cancers in men worldwide. Bi-parametric MRI (bp-MRI) and clinical variables are crucial for PCa identification and improving treatment decisions. However, this process is subjective to expert interpretations. Furthermore, most existing computer-aided diagnosis methods focus on imaging-based models, overlooking the clinical context and suffering from data scarcity, limiting their ability to learn robust representations. We propose a geometric multimodal Foundation Model (FM), named MFM-Geom, that learns representations from bp-MRI and clinical reports, encoding visual findings and information from the context of clinical variables. In the representations classification head, the approach leverages symmetric positive definite (SPD) matrices and Riemannian deep learning to integrate imaging-text representations from a biomedical multimodal FM. Using 10% of the training data, MFM-Geom outperformed baseline class token embedding-based classification (+8.3%, AUC-PR of 90.67). Generalization on external dataset confirmed the robustness of fine-tuning biomedical FM, achieving an AUC-PR of 90.6.
Paper Structure (7 sections, 3 figures, 1 table)

This paper contains 7 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Datasets
  4. Experimental Setup
  5. Evaluation and Results
  6. Conclusions
  7. Compliance with ethical standards

Figures (3)

  • Figure 1: Pipeline of the proposed geometric multimodal foundation model (MFM-Geom). Bp-MRI volumes feed independent image encoders, while clinical variables are converted into a clinical report and fed to the text encoder. The model is fine-tuned using contrastive (InfoNCE) loss on image-text class embeddings, while patch and text embeddings are used to construct a geometric SPD descriptor for classification, considering the Riemannian geometry of SPD matrices space.
  • Figure 2: Classification results varying training percentage for the three tasks, compared to different baselines.
  • Figure 3: Attention maps of the proposed method extracted from the image (A) and text (B) encoders.