How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses

TL;DR

This work tackles the challenge of automatically evaluating multi-modal LLMs for CT scan interpretation by introducing GPTRadScore, a GPT-4–based framework that decomposes AI-generated CT findings into body part, location, and lesion type and benchmarks them against clinician-ground truth. It combines a two-stage workflow (bounding-box–assisted description generation and CoT reasoning with subsequent auto-evaluation) with a fine-tuned RadFM to improve clinical descriptiveness on the DeepLesion dataset. The results show GPTRadScore correlates strongly with clinician assessments, outperforming traditional NLG metrics and revealing model-specific strengths, particularly for GPT-4V and Gemini Pro Vision, while also demonstrating substantial gains from domain-specific RadFM fine-tuning. The study provides a path toward scalable, clinically meaningful evaluation of radiology descriptions and plans to release expert-annotated benchmarks to support future research in this area.

Abstract

Automatically interpreting CT scans can ease the workload of radiologists. However, this is challenging mainly due to the scarcity of adequate datasets and reference standards for evaluation. This study aims to bridge this gap by introducing a novel evaluation framework, named ``GPTRadScore''. This framework assesses the capabilities of multi-modal LLMs, such as GPT-4 with Vision (GPT-4V), Gemini Pro Vision, LLaVA-Med, and RadFM, in generating descriptions for prospectively-identified findings. By employing a decomposition technique based on GPT-4, GPTRadScore compares these generated descriptions with gold-standard report sentences, analyzing their accuracy in terms of body part, location, and type of finding. Evaluations demonstrated a high correlation with clinician assessments and highlighted its potential over traditional metrics, such as BLEU, METEOR, and ROUGE. Furthermore, to contribute to future studies, we plan to release a benchmark dataset annotated by clinicians. Using GPTRadScore, we found that while GPT-4V and Gemini Pro Vision fare better, their performance revealed significant areas for improvement, primarily due to limitations in the dataset used for training these models. To demonstrate this potential, RadFM was fine-tuned and it resulted in significant accuracy improvements: location accuracy rose from 3.41\% to 12.8\%, body part accuracy from 29.12\% to 53\%, and type accuracy from 9.24\% to 30\%, thereby validating our hypothesis.

Figures (4)

• Figure 1: "GPTRadScore" framework for the auto-evaluation of LLM descriptions of CT-based findings. CT slices with outlined lesions were fed to vision-based LLMs that generated a description of the finding. They were then evaluated against the gold-standard sentences by a clinician, with NLG metrics, and auto-evaluation with GPT-4.
• Figure 2: Comparison of the responses from multi-modal LLMs (using CoT reasoning) for a renal cyst in the right kidney. Red, blue and purple fonts denote incorrect, correct, and uncertain descriptions respectively.
• Figure 3: Heatmap of pairwise Pearson’s Correlation Coefficient among various grading scores; traditional metrics, Clinician evaluations and GPTRadScore for Gemini Pro Vision, GPT-4V, LLaVA-Med, RadFM, and RadFM (FT). Color intensity indicates the strength of correlation, with darker shades representing higher correlation.
• Figure 4: Comparison of results of abnormality characterization by GPT-4V, Gemini Pro Vision, LLaVA-Med, RadFM, and RadFM (FT) with bounding boxes (bbox) vs. without bounding boxes (w/o bbox). Color mapping = {orange: 'Incorrect', beige: 'Partially Correct', teal: 'Correct', white: 'Not Applicable'}. $x$-axis denotes scores $\{\ x\in \mathbb{R} \ | \ 0 < x < 1\ \}$, $N$ = 500 samples.