MedS$^3$: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervision

TL;DR

MedS$^3$ introduces a self-evolving, slow-thinking framework that equips compact medical LLMs with robust long-chain reasoning through MCTS-generated, rule-verified trajectories and a soft dual-sided Process Reward Model. By coupling two iterative training streams—policy fine-tuning with SFT/DPO and PRM refinement with soft labels—the approach achieves strong performance across 11 clinical benchmarks and demonstrates notable generalization to out-of-domain tasks. Key contributions include a novel PRM learning objective, a two-iteration self-evolution pipeline, and empirical evidence of improved interpretability and reduced hallucinations. The work suggests that well-structured, step-wise supervision can enable smaller, deployable medical models to match or exceed larger proprietary systems in realistic clinical reasoning settings.

Abstract

Medical language models face critical barriers to real-world clinical reasoning applications. However, mainstream efforts, which fall short in task coverage, lack fine-grained supervision for intermediate reasoning steps, and rely on proprietary systems, are still far from a versatile, credible and efficient language model for clinical reasoning usage. To this end, we propose MedS3, a self-evolving framework that imparts robust reasoning capabilities to small, deployable models. Starting with 8,000 curated instances sampled via a curriculum strategy across five medical domains and 16 datasets, we use a small base policy model to conduct Monte Carlo Tree Search (MCTS) for constructing rule-verifiable reasoning trajectories. Self-explored reasoning trajectories ranked by node values are used to bootstrap the policy model via reinforcement fine-tuning and preference learning. Moreover, we introduce a soft dual process reward model that incorporates value dynamics: steps that degrade node value are penalized, enabling fine-grained identification of reasoning errors even when the final answer is correct. Experiments on eleven benchmarks show that MedS3 outperforms the previous state-of-the-art medical model by +6.45 accuracy points and surpasses 32B-scale general-purpose reasoning models by +8.57 points. Additional empirical analysis further demonstrates that MedS3 achieves robust and faithful reasoning behavior.

Figures (14)

• Figure 1: Overview of the construction of MedS$^3$ framework. (a) MedS$^3$ utilizes a Monte-Carlo Tree Search pipeline to self-generate step-by-step reasoning paths for each instance sampled in a curriculum manner. (b) During this process, MedS$^3$ uses result simulation to obtain the rollout value for each node; (c) After obtaining the child's rollout value, MedS$^3$ executes back-propagation to enable precise value prediction from deeper layers to transfer back to shallow nodes. (d) After the exploration finishes, we use SFT and DPO to optimize the policy model and soft dual-side label to fine-tune the process reward model.
• Figure 2: Overview of the used seed datasets.
• Figure 3: Interpretability evaluation for models using synthetic data, where MedS$^3$ produces the least hallucinatory contents among other pioneering models.
• Figure 4: Scaling in (a) self-evolution iterations and (b) sampling numbers during test-time. Both the policy and PRM harvest consistent enhancement with self-evolution, and hence their cooperative system MedS$^3$ achieves a log-linear scaling rate with little saturation.
• Figure 5: Reflective response ratio of MedS$^3$ across 7 representative datasets. Both the policy and PRM are reflection-aware to perform sequential test-time scaling.