Pathology-CoT: Learning Visual Chain-of-Thought Agent from Expert Whole Slide Image Diagnosis Behavior

TL;DR

This work tackles the data bottleneck in creating agentive, explainable pathology AI by recording expert WSI viewing behavior and converting it into structured supervision to train Pathology-o3, a two-stage agent that first proposes diagnostically relevant ROIs and then reasons over them with a vision-language model. The AI Session Recorder converts raw viewing logs into discrete actions and ROI rationales, yielding the Pathology-CoT dataset (5,222 rounds from 8 pathologists across 921 sessions). Pathology-o3, powered by a library of task-specific Behavior Predictors and a central Reasoning Module, achieves state-of-the-art performance on CRC lymph node metastasis with perfect or near-perfect recall internally and strong generalization to an external LNCO2 cohort (Sweden), outperforming OpenAI o3 and other baselines. This data-centric, modular approach demonstrates a scalable path toward human-aligned, upgradeable clinical AI in pathology, with open-source data and code to enable future expansion and deployment.

Abstract

Diagnosing a whole-slide image is an interactive, multi-stage process of changing magnification and moving between fields. Although recent pathology foundation models demonstrated superior performances, practical agentic systems that decide what field to examine next, adjust magnification, and deliver explainable diagnoses are still lacking. Such limitation is largely bottlenecked by data: scalable, clinically aligned supervision of expert viewing behavior that is tacit and experience-based, not documented in textbooks or internet, and therefore absent from LLM training. Here we introduce a framework designed to address this challenge through three key breakthroughs. First, the AI Session Recorder seamlessly integrates with standard whole-slide image viewers to unobtrusively record routine navigation and convert the viewer logs into standardized behavioral commands and bounding boxes. Second, a lightweight human-in-the-loop review turns AI-drafted rationales for behavioral commands into the Pathology-CoT dataset, a form of paired "where to look" and "why it matters", enabling six-fold faster labeling compared to manual constructing such Chain-of-Thought dataset. Using this behavioral data, we build Pathology-o3, a two-stage agent that first proposes important ROIs and then performs behavior-guided reasoning. On the gastrointestinal lymph-node metastasis detection task, our method achieved 100 recall on the internal validation from Stanford Medicine and 97.6 recall on an independent external validation from Sweden, exceeding the state-of-the-art OpenAI o3 model and generalizing across backbones. To our knowledge, Pathology-CoT constitutes one of the first behavior-grounded agentic systems in pathology. Turning everyday viewer logs into scalable, expert-validated supervision, our framework makes agentic pathology practical and establishes a path to human-aligned, upgradeable clinical AI.

Figures (10)

• Figure 1: The AI Session Recorder framework for converting expert viewing behavior into agent-ready training data. a The AI Session Recorder addresses the core data challenge: raw, continuous user logs are too noisy and complex for AI to learn from. The recorder's algorithm transforms this chaotic data stream into discrete, meaningful commands (e.g., 5x-Inspect), inspired by the discrete objective lenses (e.g., 5x, 10x, 40x) on physical microscopes. b The recorder's novel data generation pipeline creates the Pathology-CoT dataset. For each captured action, an AI drafts a rationale explaining the expert's focus, which a pathologist then efficiently verifies or corrects in a human-in-the-loop workflow. c The Pathology-CoT dataset contains 5,222 conversational rounds from pathologists with diverse experience levels. d The semi-automated, human-in-the-loop workflow is highly efficient, reducing labeling time by approximately six-fold compared to manual annotation from scratch.
• Figure 2: Overview of Pathology-o3 and its performance. a Workflow: a task prompt initializes the agent, which proposes candidate regions using a behavior bank/locating module; cropped ROIs are sent to a VLM that generates per‑ROI reasoning, then summarized into an impression and final diagnosis. b Example "thinking with image” on one slide. The agent lists planned inspections, provides step‑wise descriptions for each ROI (blue boxes), and synthesizes a case‑level conclusion. Positive and negative findings are indicated for each step. c Quantitative comparison on the lymph‑node metastasis task. Pathology-o3 achieves the best overall accuracy with perfect or near‑perfect recall, outperforming strong VLM baselines. Error bars represent 95% bootstrap confidence intervals.
• Figure 3: Quantitative and qualitative results on the external LNCO2 validation cohort. a Bar charts comparing Accuracy, Precision, and Recall of Pathology-CoT with other VLM backbones. Error bars represent 95% bootstrap confidence intervals. b Qualitative examples of the model's output on two slide images, with highlighted regions and corresponding textual descriptions.
• Figure 4: Analysis of reasoning length and order. a The effect of sequential, reverse, and random order of evidence, and the impact of chain-of-thought length on performance. b Qualitative and quantitative comparison of different models' predictions against a senior pathologist's behavior.
• Figure 5: Comparison of different "thinking" modes and cost-benefit analysis. a A table comparing cost, time, and performance between high-cost reasoning model Gemini-2.5-pro and low cost. b Performance comparison of "Non-agent", "Agent guided by real behavior", and "Agent guided by learned behavior" modes across various VLMs.