Zero-Shot Multi-modal Large Language Model v.s. Supervised Deep Learning: A Comparative Study on CT-Based Intracranial Hemorrhage Subtyping

TL;DR

This study compares zero-shot multi-modal large language models (MLLMs) against traditional deep networks for intracranial hemorrhage (ICH) binary detection and subtype classification on non-contrast CT (NCCT) scans derived from RSNA data. By employing a progressive prompt design and evaluating both proprietary and open-source MLLMs alongside learning-based classifiers, the work reveals that deep networks substantially outperform MLLMs in accuracy for both binary and subtype tasks, though MLLMs offer interpretability and clinical-context benefits. The results highlight current limitations of MLLMs for 3D medical image analysis and underscore the need for targeted fine-tuning and volumetric processing to realize their potential in radiology workflows. Overall, while MLLMs enhance interpretability, the practical deployment of these models for ICH subtyping will require substantial improvements in 3D image handling and task-specific adaptation.

Abstract

Introduction: Timely identification of intracranial hemorrhage (ICH) subtypes on non-contrast computed tomography is critical for prognosis prediction and therapeutic decision-making, yet remains challenging due to low contrast and blurring boundaries. This study evaluates the performance of zero-shot multi-modal large language models (MLLMs) compared to traditional deep learning methods in ICH binary classification and subtyping. Methods: We utilized a dataset provided by RSNA, comprising 192 NCCT volumes. The study compares various MLLMs, including GPT-4o, Gemini 2.0 Flash, and Claude 3.5 Sonnet V2, with conventional deep learning models, including ResNet50 and Vision Transformer. Carefully crafted prompts were used to guide MLLMs in tasks such as ICH presence, subtype classification, localization, and volume estimation. Results: The results indicate that in the ICH binary classification task, traditional deep learning models outperform MLLMs comprehensively. For subtype classification, MLLMs also exhibit inferior performance compared to traditional deep learning models, with Gemini 2.0 Flash achieving an macro-averaged precision of 0.41 and a macro-averaged F1 score of 0.31. Conclusion: While MLLMs excel in interactive capabilities, their overall accuracy in ICH subtyping is inferior to deep networks. However, MLLMs enhance interpretability through language interactions, indicating potential in medical imaging analysis. Future efforts will focus on model refinement and developing more precise MLLMs to improve performance in three-dimensional medical image processing.

Figures (5)

• Figure 1: (a) Random samples from the utilized dataset, with hemorrhage subtypes labeled in red; (b) The designed prompt.
• Figure 2: Line chart showing the performance of various models in the recognition of ICH subtypes.
• Figure 3: Comparison of responses from six MLLMs on a random case, which illustrates a fracture in the patient's left skull due to trauma, with SDH evident in the left hemisphere and IPH in the right hemisphere.
• Figure 4: Comparison of responses from six MLLMs to two slices from different layers of the same NCCT volume.
• Figure 5: Statistical classification results of ICH subtypes across different hemorrhage volume ranges. The bar chart presents data for all positive cases, with the macro-averaged F1 score used for evaluation.