Clinical-R1: Empowering Large Language Models for Faithful and Comprehensive Reasoning with Clinical Objective Relative Policy Optimization

TL;DR

The paper targets the gap in medical LLMs where reasoning must be faithful and comprehensive, not just correct. It introduces Clinical-objective Relative Policy Optimization (CRPO), a multi-objective, rule-based RL framework that enforces structured clinical reasoning and verifiable justifications, and demonstrates its effectiveness by training Clinical-R1-3B. Extensive experiments on MedQA, MedMCQA, and MedXpertQA show that CRPO improves medical faithfulness and cognitive comprehensiveness while maintaining or boosting accuracy compared to GRPO. This work provides a scalable approach to safer clinical AI and highlights the value of verifiable RL for domain-specific reasoning.

Abstract

Recent advances in large language models (LLMs) have shown strong reasoning capabilities through large-scale pretraining and post-training reinforcement learning, demonstrated by DeepSeek-R1. However, current post-training methods, such as Grouped Relative Policy Optimization (GRPO), mainly reward correctness, which is not aligned with the multi-dimensional objectives required in high-stakes fields such as medicine, where reasoning must also be faithful and comprehensive. We introduce Clinical-Objective Relative Policy Optimization (CRPO), a scalable, multi-objective, verifiable reinforcement learning method designed to align LLM post-training with clinical reasoning principles. CRPO integrates rule-based and verifiable reward signals that jointly optimize accuracy, faithfulness, and comprehensiveness without relying on human annotation. To demonstrate its effectiveness, we train Clinical-R1-3B, a 3B-parameter model for clinical reasoning. The experiments on three benchmarks demonstrate that our CRPO substantially improves reasoning on truthfulness and completeness over standard GRPO while maintaining comfortable accuracy enhancements. This framework provides a scalable pathway to align LLM reasoning with clinical objectives, enabling safer and more collaborative AI systems for healthcare while also highlighting the potential of multi-objective, verifiable RL methods in post-training scaling of LLMs for medical domains.

Figures (6)

• Figure 1: The accuracy, medical-faithfulness, and comprehensiveness evaluation across different methods. The result is averaged over 3 datasets (MedQA, MedMCQA, and MedXpertQA). For consistency of interpretation (higher is better), the Hallucination score is reported as $(100 - \text{Hallucination})$.
• Figure 2: Overview of our Clinical-objective Relative Policy Optimization (CRPO) Design. The model is refined via on-policy CRPO with the above reward design.
• Figure 3: Accuracy Comparison with GRPO and CRPO.
• Figure 4: Accuracy Comparison with GRPO and CRPO.
• Figure 5: Response length vs training step (MedQA).