RadDiff: Describing Differences in Radiology Image Sets with Natural Language

TL;DR

RadDiff tackles the challenge of describing clinically meaningful differences between large cohorts of radiology images by introducing a multimodal, radiologist-inspired agentic system built on a proposer–ranker framework. It integrates domain-specific vision–language models, multimodal reasoning with text reports, iterative hypothesis refinement, and targeted visual search to produce high-quality, clinically grounded difference descriptions. A dedicated RadDiffBench benchmark (57 expert-validated pairs) demonstrates substantial gains over general-domain baselines, with 47.37% top-1 accuracy and up to 50.88% using expert text, highlighting the value of medical grounding. The approach proves versatile across applications including pneumonia survival analysis, age-based COVID-19 phenotype comparisons, and auditing racial differences to reveal non-biological confounders, offering a foundation for interpretable, fair, and discovery-oriented radiology AI.

Abstract

Understanding how two radiology image sets differ is critical for generating clinical insights and for interpreting medical AI systems. We introduce RadDiff, a multimodal agentic system that performs radiologist-style comparative reasoning to describe clinically meaningful differences between paired radiology studies. RadDiff builds on a proposer-ranker framework from VisDiff, and incorporates four innovations inspired by real diagnostic workflows: (1) medical knowledge injection through domain-adapted vision-language models; (2) multimodal reasoning that integrates images with their clinical reports; (3) iterative hypothesis refinement across multiple reasoning rounds; and (4) targeted visual search that localizes and zooms in on salient regions to capture subtle findings. To evaluate RadDiff, we construct RadDiffBench, a challenging benchmark comprising 57 expert-validated radiology study pairs with ground-truth difference descriptions. On RadDiffBench, RadDiff achieves 47% accuracy, and 50% accuracy when guided by ground-truth reports, significantly outperforming the general-domain VisDiff baseline. We further demonstrate RadDiff's versatility across diverse clinical tasks, including COVID-19 phenotype comparison, racial subgroup analysis, and discovery of survival-related imaging features. Together, RadDiff and RadDiffBench provide the first method-and-benchmark foundation for systematically uncovering meaningful differences in radiological data.

Figures (52)

• Figure 1: RadDiff is designed to identify the differences between two groups of radiology images. In this example, older COVID-19 patients display more findings than younger COVID-19 patients.
• Figure 2: RadDiff algorithm. To solve the challenging task of identifying differences between two large sets consisting of thousands of images, RadDiff leverages the proposer–ranker framework from VisDiff, which first generates candidate differences from subsets and then ranks them based on a saliency score reflecting differences between the full sets. RadDiff incorporates four improvements to enhance performance: (1) medical knowledge injection through domain-adapted vision-language models; (2) multimodal reasoning that integrates images with their clinical reports; (3) iterative hypothesis refinement across multiple reasoning rounds; and (4) targeted visual search that localizes and zooms in on salient regions to capture subtle findings.
• Figure 3: RadDiffBench creation and evaluation.RadDiffBench is created in two stages: we first use LLMs to propose clinically meaningful cohort pairs, and then classify images into each pair by using an LLM to categorize clinical reports as a proxy for the image labels. For evaluation, we use an LLM to assign a three-level score representing the similarity between the predicted difference and the ground-truth difference.
• Figure 4: Ablation of iterative refinement rounds. We find that iterative refinement improves performance, with the model plateauing around the third round.
• Figure 8: Prompt used for Hypothetical Differences Proposal