Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints

TL;DR

MR-RML introduces a multidimensional, rubric-oriented reward framework for medical LLMs, anchored by a 3D Dimensions-Scenarios-Disciplines standard system and a geometric projection reference constraint that regularizes reward gradients. By internalizing medical standards into a multi-dimensional reward model and coupling it with SFT cold-start and RL optimization, the approach achieves state-of-the-art results on HealthBench among open-source models and strong performance versus closed-source counterparts. Key innovations include the prompt-anchored polar decomposition for axis-based ranking, axis similarity and perpendicular deviation losses, and a data-generation pipeline that leverages synthetic samples to reduce expert annotation costs. Empirical results demonstrate improved clinical alignment, scalability across scenarios, and substantial labor-cost reductions, suggesting practical potential for deployable, standards-driven medical AI systems.

Abstract

The integration of large language models (LLMs) into medical practice offers transformative potential, yet their real-world clinical applicability remains constrained by critical alignment issues: (1) a misalignment between static evaluation benchmarks and the dynamic cognitive demands of clinical practice, (2) challenges in adapting to continuously evolving, multi-source medical standards, and (3) the limited capacity of conventional reward models to reflect nuanced, multi-dimensional medical quality criteria. To overcome these limitations, we introduce MR-RML (Multidimensional Rubric-oriented Reward Model Learning) with GPRC (Geometric Projection Reference Constraints)-a novel alignment framework that structured medical standards into a multi-perspective matrix to guide both data generation and model optimization. Our approach introduces three key innovations: (1) a medical standard system that embeds domain-specific guidelines throughout the training pipeline; (2) an independent multi-dimensional reward model that decomposes evaluation criteria, transitioning from rule-based or LLM-based scoring to internalized reward modeling for better evaluation performance; and (3) geometric projection reference constraints that translate clinical cognitive logic into mathematical regularization, aligning scoring gradients with clinical reasoning and facilitating training with synthetically generated data. Extensive evaluations on the authoritative medical benchmark Healthbench demonstrate that our method significantly boosts the performance of the base Qwen-32B model, with improvements of 45% on the full subset and 85% on the hard subset. It achieves state-of-the-art results among open-source LLMs, scoring 62.7 (full) and 44.7 (hard), while also surpassing the majority of closed-source models.

Figures (6)

• Figure 1: Geometric constraint visualization for reward modeling.
• Figure 2: Scoring with a unified reward model from multidimensions during reinforcement learning.
• Figure 3: Overall performance on HealthBench (Full) for open-sourse LLMs.
• Figure 4: Overall performance on HealthBench (Full) for closed-sourse LLMs.
• Figure 5: Performance on HealthBench Hard (high-complexity tasks) for open-sourse LLMs.