Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval

TL;DR

Content-based medical image retrieval (CBMIR) is advanced by comparing pre-trained CNNs and foundation models as feature extractors across eight MedMNIST V2 datasets, including both 2D and 3D modalities, with a focus on zero-shot retrieval using cosine similarity. The study evaluates eight feature extractors (VGG19, ResNet50, DenseNet121, EfficientNetV2M, MedCLIP, BioMedCLIP, OpenCLIP, CONCH, UNI) and reports performance with $mAP@5$, $mMV@5$, and $ACC@1/3/5$ across varying image sizes, revealing that foundation models generally outperform CNNs on 2D data while 3D results are closer. Key findings show UNI delivering top 2D performance and CONCH achieving strong 3D results, with larger image sizes offering modest gains primarily for 2D datasets. The work provides a reproducible pipeline (code available) and offers guidance for selecting feature extractors in CBMIR, while outlining avenues for improving 3D fusion and expanding foundation-model usage in medical imaging tasks, $e.g.$, $mAP@5$, $ACC@1$, and $ACC@3$ analyses across modalities.

Abstract

Medical image retrieval refers to the task of finding similar images for given query images in a database, with applications such as diagnosis support. While traditional medical image retrieval relied on clinical metadata, content-based medical image retrieval (CBMIR) depends on image features, which can be extracted automatically or semi-automatically. Many approaches have been proposed for CBMIR, and among them, using pre-trained convolutional neural networks (CNNs) is a widely utilized approach. However, considering the recent advances in the development of foundation models for various computer vision tasks, their application for CBMIR can also be investigated. In this study, we used several pre-trained feature extractors from well-known pre-trained CNNs and pre-trained foundation models and investigated the CBMIR performance on eight types of two-dimensional (2D) and three-dimensional (3D) medical images. Furthermore, we investigated the effect of image size on the CBMIR performance. Our results show that, overall, for the 2D datasets, foundation models deliver superior performance by a large margin compared to CNNs, with the general-purpose self-supervised model for computational pathology (UNI) providing the best overall performance across all datasets and image sizes. For 3D datasets, CNNs and foundation models deliver more competitive performance, with contrastive learning from captions for histopathology model (CONCH) achieving the best overall performance. Moreover, our findings confirm that while using larger image sizes (especially for 2D datasets) yields slightly better performance, competitive CBMIR performance can still be achieved even with smaller image sizes. Our codes to reproduce the results are available at: https://github.com/masih4/MedImageRetrieval.

Figures (11)

• Figure 1: The general workflow of the image retrieval method. CNN: Convolutional Neural Networks.
• Figure 2: Example images from the MedMNIST V2 dataset. For the last two images from the 3D datasets, middle slices from random samples were chosen.
• Figure 3: ACC@1 performance based on different image sizes for the BreastMNIST dataset. Res.50 = ResNet50, Dense.121 = DenseNet 121, Eff.V2M = EfficientNetV2M.
• Figure 4: ACC@1 performance based on different image sizes for the PneumoniaMNIST dataset. Res.50 = ResNet50, Dense.121 = DenseNet 121, Eff.V2M = EfficientNetV2M.
• Figure 5: ACC@1 performance based on different image sizes for the RetinaMNIST dataset. Res.50 = ResNet50, Dense.121 = DenseNet 121, Eff.V2M = EfficientNetV2M.