Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems

Yinzhu Chen, Abdine Maiga, Hossein A. Rahmani, Emine Yilmaz

TL;DR

The paper tackles the challenge of safely evaluating medical dialogue models, where subtle clinical errors can escape generic metrics. It introduces a retrieval-augmented multi-agent framework that automatically generates instance-specific rubrics grounded in authoritative medical evidence, decomposing evidence into atomic facts and interaction intents, and auditing for coverage. On HealthBench, the approach achieves a Clinical Intent Alignment of $60.12\%$ and AUROC of $0.977$, outperforming the GPT-4o baseline, with a mean score delta of $\mu_{\Delta}=8.658$ on near-miss discrimination and a downstream response refinement gain of $9.2\%$. The work provides a scalable and transparent foundation for both evaluating and improving medical LLMs by delivering interpretable rubrics and rubric-guided feedback for refinement.

Abstract

Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety. These risks are particularly challenging as they often manifest as subtle clinical errors that evade detection by generic metrics, while expert-authored fine-grained rubrics remain costly to construct and difficult to scale. In this paper, we propose a retrieval-augmented multi-agent framework designed to automate the generation of instance-specific evaluation rubrics. Our approach grounds evaluation in authoritative medical evidence by decomposing retrieved content into atomic facts and synthesizing them with user interaction constraints to form verifiable, fine-grained evaluation criteria. Evaluated on HealthBench, our framework achieves a Clinical Intent Alignment (CIA) score of 60.12%, a statistically significant improvement over the GPT-4o baseline (55.16%). In discriminative tests, our rubrics yield a mean score delta ($μ_Δ = 8.658$) and an AUROC of 0.977, nearly doubling the quality separation achieved by GPT-4o baseline (4.972). Beyond evaluation, our rubrics effectively guide response refinement, improving quality by 9.2% (from 59.0% to 68.2%). This provides a scalable and transparent foundation for both evaluating and improving medical LLMs. The code is available at https://anonymous.4open.science/r/Automated-Rubric-Generation-AF3C/.

Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems

TL;DR

The paper tackles the challenge of safely evaluating medical dialogue models, where subtle clinical errors can escape generic metrics. It introduces a retrieval-augmented multi-agent framework that automatically generates instance-specific rubrics grounded in authoritative medical evidence, decomposing evidence into atomic facts and interaction intents, and auditing for coverage. On HealthBench, the approach achieves a Clinical Intent Alignment of and AUROC of , outperforming the GPT-4o baseline, with a mean score delta of on near-miss discrimination and a downstream response refinement gain of . The work provides a scalable and transparent foundation for both evaluating and improving medical LLMs by delivering interpretable rubrics and rubric-guided feedback for refinement.

Abstract

Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety. These risks are particularly challenging as they often manifest as subtle clinical errors that evade detection by generic metrics, while expert-authored fine-grained rubrics remain costly to construct and difficult to scale. In this paper, we propose a retrieval-augmented multi-agent framework designed to automate the generation of instance-specific evaluation rubrics. Our approach grounds evaluation in authoritative medical evidence by decomposing retrieved content into atomic facts and synthesizing them with user interaction constraints to form verifiable, fine-grained evaluation criteria. Evaluated on HealthBench, our framework achieves a Clinical Intent Alignment (CIA) score of 60.12%, a statistically significant improvement over the GPT-4o baseline (55.16%). In discriminative tests, our rubrics yield a mean score delta () and an AUROC of 0.977, nearly doubling the quality separation achieved by GPT-4o baseline (4.972). Beyond evaluation, our rubrics effectively guide response refinement, improving quality by 9.2% (from 59.0% to 68.2%). This provides a scalable and transparent foundation for both evaluating and improving medical LLMs. The code is available at https://anonymous.4open.science/r/Automated-Rubric-Generation-AF3C/.
Paper Structure (51 sections, 11 equations, 4 figures, 16 tables)

This paper contains 51 sections, 11 equations, 4 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NLU Evaluation.
  4. NLG Evaluation.
  5. LLM-as-a-Judge.
  6. Rubric-based LLM-as-a-Judge.
  7. Summary and Positioning.
  8. Methodology
  9. Problem Formulation
  10. Stage 1: Retrieval & Evidence Preparation
  11. Routing Agent (Smart--Fast Strategy).
  12. Evidence Synthesis Agent.
  13. Stage 2: Dual-Track Constraint Construction
  14. Medical Fact Agent (Atomic Fact Decomposition).
  15. Interaction Intent Agent.
  16. ...and 36 more sections

Figures (4)

  • Figure 1: Retrieval-augmented multi-agent framework for medical rubric generation. The pipeline consists of three stages: (1) Retrieval and Evidence Preparation, (2) Dual-Track Constraint Construction and (3) Audit and Refinement, transforming a medical user query into a structured evaluation rubric.
  • Figure 2: Discrimination analysis on the micro-perturbed dataset: (A) Mean score difference between reference and perturbed responses, (B) outcome distribution (win/tie/lose), and (C) AUROC across rubric settings.
  • Figure 3: Dimension-wise analysis of downstream response refinement under different rubric settings, including overall performance trends and trade-offs across evaluation dimensions.
  • Figure 4: An evaluation example from HealthBench(Arora et al., 2025), where a model-generated response is graded against physician-written rubrics tailored to the specific conversation.