Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpineBench: Benchmarking Multimodal LLMs for Spinal Pathology Analysis

Chenghanyu Zhang, Zekun Li, Peipei Li, Xing Cui, Shuhan Xia, Weixiang Yan, Yiqiao Zhang, Qianyu Zhuang

TL;DR

SpineBench addresses the lack of spine-specific benchmarks for multimodal LLMs by creating a large-scale spine VQA dataset with two clinical tasks and hard distractors. It standardizes four open spine datasets into 40,263 disease-labeled images and 24,615 lesion-labeled images, yielding 64,878 VQA pairs; evaluation is performed on 12 MLLMs across general-purpose and medical-specialized models. The results show that state-of-the-art MLLMs struggle with fine-grained spinal pathology tasks, achieving low accuracy and revealing gaps in visual semantic understanding and medical reasoning. The work provides a practical benchmark and evaluation framework to drive future improvements in spine-domain MLLMs and clinical applicability.

Abstract

With the increasing integration of Multimodal Large Language Models (MLLMs) into the medical field, comprehensive evaluation of their performance in various medical domains becomes critical. However, existing benchmarks primarily assess general medical tasks, inadequately capturing performance in nuanced areas like the spine, which relies heavily on visual input. To address this, we introduce SpineBench, a comprehensive Visual Question Answering (VQA) benchmark designed for fine-grained analysis and evaluation of MLLMs in the spinal domain. SpineBench comprises 64,878 QA pairs from 40,263 spine images, covering 11 spinal diseases through two critical clinical tasks: spinal disease diagnosis and spinal lesion localization, both in multiple-choice format. SpineBench is built by integrating and standardizing image-label pairs from open-source spinal disease datasets, and samples challenging hard negative options for each VQA pair based on visual similarity (similar but not the same disease), simulating real-world challenging scenarios. We evaluate 12 leading MLLMs on SpineBench. The results reveal that these models exhibit poor performance in spinal tasks, highlighting limitations of current MLLM in the spine domain and guiding future improvements in spinal medicine applications. SpineBench is publicly available at https://zhangchenghanyu.github.io/SpineBench.github.io/.

SpineBench: Benchmarking Multimodal LLMs for Spinal Pathology Analysis

TL;DR

SpineBench addresses the lack of spine-specific benchmarks for multimodal LLMs by creating a large-scale spine VQA dataset with two clinical tasks and hard distractors. It standardizes four open spine datasets into 40,263 disease-labeled images and 24,615 lesion-labeled images, yielding 64,878 VQA pairs; evaluation is performed on 12 MLLMs across general-purpose and medical-specialized models. The results show that state-of-the-art MLLMs struggle with fine-grained spinal pathology tasks, achieving low accuracy and revealing gaps in visual semantic understanding and medical reasoning. The work provides a practical benchmark and evaluation framework to drive future improvements in spine-domain MLLMs and clinical applicability.

Abstract

With the increasing integration of Multimodal Large Language Models (MLLMs) into the medical field, comprehensive evaluation of their performance in various medical domains becomes critical. However, existing benchmarks primarily assess general medical tasks, inadequately capturing performance in nuanced areas like the spine, which relies heavily on visual input. To address this, we introduce SpineBench, a comprehensive Visual Question Answering (VQA) benchmark designed for fine-grained analysis and evaluation of MLLMs in the spinal domain. SpineBench comprises 64,878 QA pairs from 40,263 spine images, covering 11 spinal diseases through two critical clinical tasks: spinal disease diagnosis and spinal lesion localization, both in multiple-choice format. SpineBench is built by integrating and standardizing image-label pairs from open-source spinal disease datasets, and samples challenging hard negative options for each VQA pair based on visual similarity (similar but not the same disease), simulating real-world challenging scenarios. We evaluate 12 leading MLLMs on SpineBench. The results reveal that these models exhibit poor performance in spinal tasks, highlighting limitations of current MLLM in the spine domain and guiding future improvements in spinal medicine applications. SpineBench is publicly available at https://zhangchenghanyu.github.io/SpineBench.github.io/.
Paper Structure (19 sections, 4 equations, 6 figures, 3 tables)

This paper contains 19 sections, 4 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal LLMs in medical domains
  4. Medical Benchmarks
  5. Spinal Multimodal Datasets
  6. SpineBench
  7. Data Collection and Standardization
  8. VQA Pair Construction
  9. Diverse Question Generation.
  10. Image Similarity-based Distractor Option Sampling.
  11. Multi-Label Lesion Localization Task.
  12. Data Split
  13. Evaluation Metrics
  14. Experiments
  15. Experimental Details
  16. ...and 4 more sections

Figures (6)

  • Figure 1: Disease distribution in SpineBench.
  • Figure 2: SpineBench is constructed with two steps: benchmark construction and standardization (upper), and VQA pairs generation (lower).
  • Figure 3: Disease distribution of the evaluation set.
  • Figure 4: Model performance on Spinal Lesion Localization.
  • Figure 5: MLLM Reasoning Average Scores (Max score: 5) from medical experts. Most models scored below 4, indicating limited capability in spatial pathological reasoning.
  • ...and 1 more figures