Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models

Jakob Krogh Petersen, Valdemar Licht, Mads Nielsen, Asbjørn Munk

TL;DR

This work demonstrates CLIP-style alignment between native 3D brain MRI and tabular data by training domain-specific 3D MRI encoders pretrained with foundation-model techniques. It introduces an embedding accumulation strategy to scale negative samples across multiple batches, enabling stable training with limited 3D data. The approach yields meaningful zero-shot classification and retrieval signals, with Swin-T as a strong image encoder and BERT-based tabular encoding, while highlighting the challenges of zero-shot image retrieval on very small datasets. The study provides code and model weights, underscoring practical potential for domain-specific multi-modal alignment in medicine despite data scarcity.

Abstract

Multi-modal models require aligned, shared embedding spaces. However, common CLIP-based approaches need large amounts of samples and do not natively support 3D or tabular data, both of which are crucial in the medical domain. To address these issues, we revisit CLIP-style alignment by training a domain-specific 3D foundation model as an image encoder and demonstrate that modality alignment is feasible with only 62 MRI scans. Our approach is enabled by a simple embedding accumulation strategy required for training in 3D, which scales the amount of negative pairs across batches in order to stabilize training. We perform a thorough evaluation of various design choices, including the choice of backbone and loss functions, and evaluate the proposed methodology on zero-shot classification and image-retrieval tasks. While zero-shot image-retrieval remains challenging, zero-shot classification results demonstrate that the proposed approach can meaningfully align the representations of 3D MRI with tabular data.

Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models

TL;DR

This work demonstrates CLIP-style alignment between native 3D brain MRI and tabular data by training domain-specific 3D MRI encoders pretrained with foundation-model techniques. It introduces an embedding accumulation strategy to scale negative samples across multiple batches, enabling stable training with limited 3D data. The approach yields meaningful zero-shot classification and retrieval signals, with Swin-T as a strong image encoder and BERT-based tabular encoding, while highlighting the challenges of zero-shot image retrieval on very small datasets. The study provides code and model weights, underscoring practical potential for domain-specific multi-modal alignment in medicine despite data scarcity.

Abstract

Multi-modal models require aligned, shared embedding spaces. However, common CLIP-based approaches need large amounts of samples and do not natively support 3D or tabular data, both of which are crucial in the medical domain. To address these issues, we revisit CLIP-style alignment by training a domain-specific 3D foundation model as an image encoder and demonstrate that modality alignment is feasible with only 62 MRI scans. Our approach is enabled by a simple embedding accumulation strategy required for training in 3D, which scales the amount of negative pairs across batches in order to stabilize training. We perform a thorough evaluation of various design choices, including the choice of backbone and loss functions, and evaluate the proposed methodology on zero-shot classification and image-retrieval tasks. While zero-shot image-retrieval remains challenging, zero-shot classification results demonstrate that the proposed approach can meaningfully align the representations of 3D MRI with tabular data.
Paper Structure (17 sections, 2 figures, 4 tables)

This paper contains 17 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Foundation Model Encoders
  4. Encoding 3D MRI using domain-specific pretraining
  5. Encoding tabular data
  6. Effect of using foundation model encoders
  7. Computing the CLIP-loss across batches
  8. Obtaining stable training in 3D
  9. Experimental setup
  10. Data
  11. Hyperparameters and experimental details
  12. Evaluation Tasks
  13. Results & Discussion
  14. Discussion
  15. Conclusion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: t-SNE tsne visualizations of MRI scans and tabular data in the joint embedding space before and after CLIP training. Input data is the test set, with augmentations applied to increase sample size. Best viewed in color.
  • Figure 2: Pseudo code showing how accumulated loss over $N$ batches is calculated. The embedding accumulation strategy computes embeddings for all input in $N$ batches, and for each batch uses negative examples from all batches to compute the loss and update the model.