MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis

TL;DR

MM-NeuroOnco is introduced, a large-scale multimodal benchmark and instruction-tuning dataset for brain tumor MRI understanding and a multi-model collaborative pipeline for automated medical information completion and quality control, enabling the generation of diagnosis-related semantics beyond mask-only annotations.

Abstract

Accurate brain tumor diagnosis requires models to not only detect lesions but also generate clinically interpretable reasoning grounded in imaging manifestations, yet existing public datasets remain limited in annotation richness and diagnostic semantics. To bridge this gap, we introduce MM-NeuroOnco, a large-scale multimodal benchmark and instruction-tuning dataset for brain tumor MRI understanding, consisting of 24,726 MRI slices from 20 data sources paired with approximately 200,000 semantically enriched multimodal instructions spanning diverse tumor subtypes and imaging modalities. To mitigate the scarcity and high cost of diagnostic semantic annotations, we develop a multi-model collaborative pipeline for automated medical information completion and quality control, enabling the generation of diagnosis-related semantics beyond mask-only annotations. Building upon this dataset, we further construct MM-NeuroOnco-Bench, a manually annotated evaluation benchmark with a rejection-aware setting to reduce biases inherent in closed-ended question formats. Evaluation across ten representative models shows that even the strongest baseline, Gemini 3 Flash, achieves only 41.88% accuracy on diagnosis-related questions, highlighting the substantial challenges of multimodal brain tumor diagnostic understanding. Leveraging MM-NeuroOnco, we further propose NeuroOnco-GPT, which achieves a 27% absolute accuracy improvement on diagnostic questions following fine-tuning. This result demonstrates the effectiveness of our dataset and benchmark in advancing clinically grounded multimodal diagnostic reasoning. Code and dataset are publicly available at: https://github.com/gfnnnb/MM-NeuroOnco

Figures (20)

• Figure 1: Overview of MM-NeuroOnco. The benchmark curates 24,726 slices from 20 diverse datasets. Standard diagnostic labels are augmented with fine-grained semantic attributes to construct explicit Chain-of-Thought (CoT) reasoning. This structure enables a multi-grained evaluation paradigm, covering both holistic open-ended diagnosis and targeted closed-ended Question Answering across multiple clinical categories.
• Figure 2: The MM-NeuroOnco dataset curation pipeline, which consists of four sequential steps transforming raw MRI data into semantically enriched multimodal instructions.
• Figure 3: Radar chart comparing the performance of the base model and different fine-tuning strategies on MM-NeuroOnco-Bench.
• Figure 4: Statistical distributions of the MM-NeuroOnco dataset.
• Figure 5: Modality distribution of the MM-NeuroOnco dataset and benchmark.The figure compares the proportional distributions of different MRI modalities in the dataset and the benchmark.