PathAgent: Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoning

TL;DR

This work tackles the opacity of whole-slide pathology analysis by introducing PathAgent, a training-free LLM-based agent that mimics pathologists’ iterative reasoning. It combines three components—Navigator for RoI search, Perceptor for morphology cues, and Executor for multi-step reasoning—to produce an explicit chain-of-thought and interpretable predictions across multi-scale WSIs. Across five challenging datasets, PathAgent demonstrates strong zero-shot generalization and surpasses task-specific baselines in both open-ended and closed-ended VQA tasks, with quantitative and qualitative evidence corroborated by pathologist collaboration. The framework offers a practical, scalable route to clinically grounded, transparent WSI analysis and interactive AI-assisted diagnosis.

Abstract

Analyzing whole-slide images (WSIs) requires an iterative, evidence-driven reasoning process that parallels how pathologists dynamically zoom, refocus, and self-correct while collecting the evidence. However, existing computational pipelines often lack this explicit reasoning trajectory, resulting in inherently opaque and unjustifiable predictions. To bridge this gap, we present PathAgent, a training-free, large language model (LLM)-based agent framework that emulates the reflective, stepwise analytical approach of human experts. PathAgent can autonomously explore WSI, iteratively and precisely locating significant micro-regions using the Navigator module, extracting morphology visual cues using the Perceptor, and integrating these findings into the continuously evolving natural language trajectories in the Executor. The entire sequence of observations and decisions forms an explicit chain-of-thought, yielding fully interpretable predictions. Evaluated across five challenging datasets, PathAgent exhibits strong zero-shot generalization, surpassing task-specific baselines in both open-ended and constrained visual question-answering tasks. Moreover, a collaborative evaluation with human pathologists confirms PathAgent's promise as a transparent and clinically grounded diagnostic assistant.

Figures (6)

• Figure 1: Illustration of current multi-modal CPath models. (a) CLIP-style Image Captioning models are bound by pre-defined captioning templates. (b) VLMs are limited by a limited receptive field. (c) Patch-aggregated LLMs lack sufficient reasoning evidence. (d) PathAgent emulates the reflective, step-wise thinking of pathologists while performing analysis.
• Figure 2: Overview of the proposed PathAgent. Given an input WSI, PathAgent iteratively collects visual evidence and aggregates key analytic information to generate interpretable results. The process is completed by Navigator, Perceptor, and Executor, where the Executor serves as the central module orchestrating all analytic actions. "Mag" means magnification and "Loc" is the abbreviation of location.
• Figure 3: Case Study of a close-ended question in WSI-VQA. PathAgent accurately locates patches based on the question and identifies the missing information in the first iteration. Then PathAgent provides supplementary descriptions by zooming in on the patch area, filling the gaps, and thus providing logical reasoning and answers. The key cues found by PathAgent are in yellow.
• Figure 4: Impact of initial patch proportion on WSI-VQA accuracy.
• Figure 5: Case study of an open-ended question in the WSI-VQA dataset. Here, PathAgent accurately locates the patches and responses the correct answer in the first iteration, while the WSI-VQA and SlideChat methods provide incorrect results. It is worth noting that GPT-4o refused to provide an answer when it only received a thumbnail, but after being provided with the patches selected by PathAgent, GPT-4o is able to answer correctly. More case analyses are in supplementary materials.