Route, Retrieve, Reflect, Repair: Self-Improving Agentic Framework for Visual Detection and Linguistic Reasoning in Medical Imaging

TL;DR

Medical image analysis with large vision-language models suffers from brittle reasoning and weak spatial grounding when treated as a single-pass, end-to-end system. The authors propose R^4, a four-agent loop (Router, Retriever, Reflector, Repairer) that conditions task configuration on patient context, retrieves exemplar priors, critiques drafts for clinical errors, and iteratively repairs both narrative reports and bounding boxes, all without gradient-based fine-tuning. A persistent exemplar memory enables few-shot context adaptation, improving reliability and localization across chest X-ray tasks, with reported increases in LLM-as-a-Judge scores by roughly $+1.7$ to $+2.5$ and mAP$_{50}$ by $+2.5$ to $+3.5$ absolute points over strong single-VLM baselines. The approach demonstrates meaningful gains in clinical fidelity and spatial grounding, suggesting a practical path to deployable, self-improving multimodal radiology tools; code is released at the provided repository URL.

Abstract

Medical image analysis increasingly relies on large vision-language models (VLMs), yet most systems remain single-pass black boxes that offer limited control over reasoning, safety, and spatial grounding. We propose R^4, an agentic framework that decomposes medical imaging workflows into four coordinated agents: a Router that configures task- and specialization-aware prompts from the image, patient history, and metadata; a Retriever that uses exemplar memory and pass@k sampling to jointly generate free-text reports and bounding boxes; a Reflector that critiques each draft-box pair for key clinical error modes (negation, laterality, unsupported claims, contradictions, missing findings, and localization errors); and a Repairer that iteratively revises both narrative and spatial outputs under targeted constraints while curating high-quality exemplars for future cases. Instantiated on chest X-ray analysis with multiple modern VLM backbones and evaluated on report generation and weakly supervised detection, R^4 consistently boosts LLM-as-a-Judge scores by roughly +1.7-+2.5 points and mAP50 by +2.5-+3.5 absolute points over strong single-VLM baselines, without any gradient-based fine-tuning. These results show that agentic routing, reflection, and repair can turn strong but brittle VLMs into more reliable and better grounded tools for clinical image interpretation. Our code can be found at: https://github.com/faiyazabdullah/MultimodalMedAgent

Figures (3)

• Figure 1: Overview of the proposed R$^4$ agentic framework. The chest X-ray and its accompanying text is used as input. Patient medical history and metadata are then passed to the Router Agent (1), which selects a task configuration and specialized LLM to handle the case. The Retriever Agent (2) queries the LLM knowledge base with $k$ retrieved exemplars to generate an initial report $I$ together with an initial bounding box $B$ in normalized coordinates $(x_1,x_2,y_1,y_2)$. The Reflect Agent (3) critiques $I$ and $B$ under targeted clinical and safety constraints, producing an issue set $I$. The Repair Agent (4) updates the draft report and bounding box using these constraints, consults the cache memory for similar exemplars, and outputs a final report and refined bounding box that are fed back into the cache for future retrieval.
• Figure 2: Pass@k ablation on report quality and localization for three R$^4$Agent backbones. Left: Overall Score LLM-as-a-Judge (Ovrl). Right: localization mAP$_{50}$.
• Figure 3: Pass@k qualitative example (backbone: Gemini-2.5-Flash). Report text and bounding-box localization improve from Pass@1 to Pass@3.