Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench

TL;DR

PET-Bench reveals a functional perception gap in multimodal large language models when interpreting PET images, showing that models pretrained on structural anatomy struggle to quantify tracer biodistribution and molecular kinetics. The authors introduce Atomic Visual Alignment (AVA), a hierarchical fine-tuning approach that enforces low-level perceptual grounding (Levels 1–4) before high-level disease diagnosis, and PET-Bench as a large-scale, multi-site PET benchmark with a five-level VQA taxonomy. They show a Chain-of-Thought (CoT) hallucination trap where fluent reasoning can be ungrounded in the absence of proper visual grounding, and demonstrate that AVA substantially improves diagnostic accuracy and reliability, including gains up to 14–15 percentage points on several models. This work provides a principled framework and dataset to develop safer, visually grounded MLLMs for functional imaging, with implications for PET, SPECT, and other functional modalities.

Abstract

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in tasks such as abnormality detection and report generation for anatomical modalities, their capability in functional imaging remains largely unexplored. In this work, we identify and quantify a fundamental functional perception gap: the inability of current vision encoders to decode functional tracer biodistribution independent of morphological priors. Identifying Positron Emission Tomography (PET) as the quintessential modality to investigate this disconnect, we introduce PET-Bench, the first large-scale functional imaging benchmark comprising 52,308 hierarchical QA pairs from 9,732 multi-site, multi-tracer PET studies. Extensive evaluation of 19 state-of-the-art MLLMs reveals a critical safety hazard termed the Chain-of-Thought (CoT) hallucination trap. We observe that standard CoT prompting, widely considered to enhance reasoning, paradoxically decouples linguistic generation from visual evidence in PET, producing clinically fluent but factually ungrounded diagnoses. To resolve this, we propose Atomic Visual Alignment (AVA), a simple fine-tuning strategy that enforces the mastery of low-level functional perception prior to high-level diagnostic reasoning. Our results demonstrate that AVA effectively bridges the perception gap, transforming CoT from a source of hallucination into a robust inference tool and improving diagnostic accuracy by up to 14.83%. Code and data are available at https://github.com/yezanting/PET-Bench.

Figures (6)

• Figure 1: Illustration of the two critical failure modes identified in MLLMs when applied to functional imaging. Functional Perception Gap: While current SOTA models achieve high performance on structural imaging benchmarks (the results from Lingshu test data), their zero-shot diagnostic accuracy on PET drops significantly. The CoT Hallucination Trap: Attempting to bridge this gap via standard CoT prompting paradoxically degrades reliability. Without domain-specific visual grounding, models generate linguistically plausible but factually ungrounded rationales, leading to high-confidence misdiagnoses.
• Figure 2: Overview of the PET-Bench framework. PET-Bench is the first large-scale benchmark designed to evaluate functional imaging capabilities, aggregating 52,308 QA pairs from 9,732 studies across 8 international centers. Unlike generic VQA datasets, the benchmark employs a five-level hierarchical taxonomy mirroring the nuclear medicine interpretation workflow: progressing from atomic perception to lesion detection, and finally to disease diagnosis. This structure allows for explicitly decoupling perceptual failures from reasoning deficits.
• Figure 3: Statistics of the PET-Bench dataset. The central sunburst chart illustrates the sample volume across the five hierarchical tasks, while outer plots detail class-wise breakdowns. The distribution reflects a deliberate design choice: large-scale data is utilized for low-level perceptual tasks to ensure robust feature learning, whereas the high-level disease diagnosis task ($N=471$) prioritizes label precision, incorporating only cases with biopsy-verified or clinically definitive outcomes.
• Figure 4: Schematic workflow of the proposed CoT prompting and evaluation protocol. The six-step clinical CoT prompt instructs the model to sequentially analyze tracer type, physiological uptake, image quality, and abnormalities before concluding a diagnosis. To quantify the CoT Hallucination Trap, an auxiliary expert LLM evaluator assesses the reasoning quality across four dimensions (Logical Coherence, Medical Accuracy, Completeness and Depth) independent of the final answer correctness.
• Figure 5: Qualitative visualization of hierarchical tasks in PET-Bench (Levels 1--4). The figure displays representative failures and successes across different models. Note that generalist models often struggle with domain-specific concepts, such as distinguishing FDG from FAPI (Level 1) or identifying noise artifacts (Level 2), highlighting the necessity of the proposed atomic visual alignment.