Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learning

TL;DR

This work introduces Med-TIV, a tool-integrated, agentic reinforcement learning framework for medical reasoning verification. By enabling iterative external knowledge retrieval during evaluation and using a trace-level supervision signal, Med-TIV grounds verification in evidence and provides interpretable critiques instead of opaque scalar scores. The approach combines an adaptive curriculum and self-bootstrapping RL to achieve strong verification performance across four medical benchmarks, while substantially reducing the sampling budget compared to prior reward-model baselines. Empirical results show Med-TIV enables smaller generators to rival larger models, with robust generalization across base architectures and test-time search strategies, highlighting its potential for safer, more reliable medical AI deployment.

Abstract

Large language models have achieved strong performance on medical reasoning benchmarks, yet their deployment in clinical settings demands rigorous verification to ensure factual accuracy. While reward models offer a scalable approach for reasoning trace verification, existing methods face two limitations: they produce only scalar reward values without explicit justification, and they rely on single-pass retrieval that precludes adaptive knowledge access as verification unfolds. We introduce $\method$, an agentic framework that addresses these limitations by training medical reasoning verifiers to iteratively query external medical corpora during evaluation. Our approach combines tool-augmented verification with an iterative reinforcement learning paradigm that requires only trace-level supervision, alongside an adaptive curriculum mechanism that dynamically adjusts training data distribution. Across four medical reasoning benchmarks, $\method$ achieves substantial gains over existing methods, improving MedQA accuracy by 23.5% and MedXpertQA by 32.0% relative to the base generator in particular. Crucially, $\method$ demonstrates an $\mathbf{8\times}$ reduction in sampling budget requirement compared to prior reward model baselines. These findings establish that grounding verification in dynamically retrieved evidence offers a principled path toward more reliable medical reasoning systems.

Figures (4)

• Figure 1: Comparison of medical reasoning verification paradigms. Text-based judges rely on parametric knowledge and may validate erroneous reasoning, while tool-integrated judges dynamically retrieve evidence to ground their judgments.
• Figure 2: Overview of Med-TIV.Left: Tool-integrated verification iteratively analyzes reasoning traces, formulates search queries, and retrieves medical evidence before producing correctness judgments. Middle: Curriculum formulation filters trivial and impossible instances, retaining boundary cases for RL training. Right: At inference time, the verifier evaluates candidate medical reasoning traces generated by a frozen model and final answers are selected via weighted self-consistency.
• Figure 3: Test-time scaling analysis across three medical reasoning benchmarks. Each plot shows accuracy versus sampling budget $N \in \{1, 2, 4, 8, 16, 32\}$ for four baselines. $\texttt{Med-TIV}\xspace$ consistently outperforms baselines across all sampling budgets and benchmarks.
• Figure 4: Ablation on base model selection and training iterations.