Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

Katarina Trojachanec Dineva, Stefan Andonov, Ilinka Ivanoska, Ivan Kitanovski, Sasho Gramatikov, Tamara Kostova, Monika Simjanoska Misheva, Kostadin Mishev

Abstract

Recent advances in multimodal large language models enable new possibilities for image-based decision support. However, their reliability and operational trade-offs in neuroimaging remain insufficiently understood. We present a comprehensive benchmarking study of vision-enabled large language models for 2D neuroimaging using curated MRI and CT datasets covering multiple sclerosis, stroke, brain tumors, other abnormalities, and normal controls. Models are required to generate multiple outputs simultaneously, including diagnosis, diagnosis subtype, imaging modality, specialized sequence, and anatomical plane. Performance is evaluated across four directions: discriminative classification with abstention, calibration, structured-output validity, and computational efficiency. A multi-phase framework ensures fair comparison while controlling for selection bias. Across twenty frontier multimodal models, the results show that technical imaging attributes such as modality and plane are nearly solved, whereas diagnostic reasoning, especially subtype prediction, remains challenging. Tumor classification emerges as the most reliable task, stroke is moderately solvable, while multiple sclerosis and rare abnormalities remain difficult. Few-shot prompting improves performance for several models but increases token usage, latency, and cost. Gemini-2.5-Pro and GPT-5-Chat achieve the strongest overall diagnostic performance, while Gemini-2.5-Flash offers the best efficiency-performance trade-off. Among open-weight architectures, MedGemma-1.5-4B demonstrates the most promising results, as under few-shot prompting, it approaches the zero-shot performance of several proprietary models, while maintaining perfect structured output. These findings provide practical insights into performance, reliability, and efficiency trade-offs, supporting standardized evaluation of multimodal LLMs in neuroimaging.

NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

Abstract

Recent advances in multimodal large language models enable new possibilities for image-based decision support. However, their reliability and operational trade-offs in neuroimaging remain insufficiently understood. We present a comprehensive benchmarking study of vision-enabled large language models for 2D neuroimaging using curated MRI and CT datasets covering multiple sclerosis, stroke, brain tumors, other abnormalities, and normal controls. Models are required to generate multiple outputs simultaneously, including diagnosis, diagnosis subtype, imaging modality, specialized sequence, and anatomical plane. Performance is evaluated across four directions: discriminative classification with abstention, calibration, structured-output validity, and computational efficiency. A multi-phase framework ensures fair comparison while controlling for selection bias. Across twenty frontier multimodal models, the results show that technical imaging attributes such as modality and plane are nearly solved, whereas diagnostic reasoning, especially subtype prediction, remains challenging. Tumor classification emerges as the most reliable task, stroke is moderately solvable, while multiple sclerosis and rare abnormalities remain difficult. Few-shot prompting improves performance for several models but increases token usage, latency, and cost. Gemini-2.5-Pro and GPT-5-Chat achieve the strongest overall diagnostic performance, while Gemini-2.5-Flash offers the best efficiency-performance trade-off. Among open-weight architectures, MedGemma-1.5-4B demonstrates the most promising results, as under few-shot prompting, it approaches the zero-shot performance of several proprietary models, while maintaining perfect structured output. These findings provide practical insights into performance, reliability, and efficiency trade-offs, supporting standardized evaluation of multimodal LLMs in neuroimaging.

Paper Structure

This paper contains 25 sections, 12 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models in Medicine
  4. Neuroimaging Models and Neuro-Specific Benchmarks
  5. Materials and Methods
  6. Benchmark Dataset
  7. Models
  8. Experimental Setup - Benchmark Design
  9. Phase 0: Experimental Setup Development and Locking (Screening Pool)
  10. Phase 1: Screening Evaluation and Early Model Filtering
  11. Phase 2: Development-Scale Confirmation
  12. Phase 3: Final Evaluation with Zero-shot and Few-shot Prompting
  13. Specification of the Prompt
  14. Evaluation
  15. Evaluation
  16. ...and 10 more sections

Figures (12)

  • Figure 1: Overview of the evaluation setup
  • Figure 2: Illustrative overview of selected output fields and evaluation dimensions assessed in NeuroVLM-Bench. The stacked bars summarize representative metrics capturing diagnostic discrimination (Diagnosis F1 and DiagnosisDetailed F1), imaging attribute recognition (Modality F1, SpecializedSequence F1, Plane F1), output reliability (Schema Validity), uncertainty behavior (Abstention Rate), and confidence calibration (ECE). This visualization provides a high-level summary of relative model behavior and is not intended as a complete performance ranking.
  • Figure 3: Overview of the data origin and class composition
  • Figure 4: Sunburst visualization of the benchmark’s output fields, illustrating the distribution of samples across diagnostic class, diagnostic subclass, and imaging modality.
  • Figure 5: Distribution of imaging-specific output fields in the benchmark.
  • ...and 7 more figures