AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning

TL;DR

The paper tackles the problem of reliably evaluating clinically correct and diagnostically faithful medical imaging reports. It introduces AgentsEval, a multi-agent stream reasoning framework that decomposes evaluation into interpretable steps—base criteria generation, evidence extraction, GT/prediction alignment, and criterion-wise scoring—producing explicit reasoning traces. A multi-domain perturbation benchmark across five datasets is constructed to test semantic fidelity and clinical reasoning under paraphrase and factual distortions. Experiments show that AgentsEval provides clinically aligned, robust, and interpretable evaluations that outperform traditional lexical metrics and single-agent baselines, even under challenging perturbations, signaling a step toward trustworthy integration of LLMs in clinical report assessment and development.

Abstract

Evaluating the clinical correctness and reasoning fidelity of automatically generated medical imaging reports remains a critical yet unresolved challenge. Existing evaluation methods often fail to capture the structured diagnostic logic that underlies radiological interpretation, resulting in unreliable judgments and limited clinical relevance. We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists. By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback. We also construct a multi-domain perturbation-based benchmark covering five medical report datasets with diverse imaging modalities and controlled semantic variations. Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations. This framework represents a step toward transparent and clinically grounded assessment of medical report generation systems, fostering trustworthy integration of large language models into clinical practice.

Figures (6)

• Figure 1: Challenges in evaluating reports generated by large models and vision–language models. Traditional NLP metrics inadequately capture clinical correctness, rewarding surface similarity instead of diagnostic accuracy. In contrast, multi-agent collaboration enables clinically grounded evaluation of medical text reports. Yellow highlights indicate key findings consistent with the ground truth (GT), gray highlights denote factual discrepancies, and blue highlights represent synonymous yet correct expressions.
• Figure 2: Detailed descriptions of different Agents: Each Agent displays its input and output above, with a simple illustration of its functionality below.
• Figure 3: Per-sample metric trends on MedVAL-Bench (a) and RadEvalX (b). Light curves: raw values; dark curves: smoothed. Black dashed line: normalized clinical errors. Red boxes: samples with inconsistent traditional metrics.
• Figure 4: Illustrative case study showing metric responses to synonymous (A2) and semantic inversion (B2) rewrites of a clinical report. The failure of traditional metrics is clearly evident.
• Figure 5: Metric sensitivity to clinical errors and linguistic paraphrases on the ReXErr-v1 dataset. Each subplot shows the smoothed per-sample scores (window is 15) for four report versions: the clinically erroneous set (3 Error) and three synonym-rewritten paraphrases (A1–A3).