MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models

TL;DR

MedDialogRubrics introduces a privacy-preserving benchmark for multi-turn medical consultations, incorporating 5,200 synthetic cases and over 60,000 expert-refined rubrics to evaluate diagnostic reasoning and information gathering in LLMs. It combines a deterministic Patient Agent with a dynamic guidance loop and an EB M-grounded rubric-generation pipeline that employs reject sampling and expert annotation to create high-quality evaluation criteria. The framework uses a calibrated LLM-as-judge setup with ensemble strategies to align automated scoring with human judgments, revealing substantial gaps in current models’ dialogue management and information-seeking capabilities. Overall, the work provides a scalable, reproducible platform that emphasizes process-oriented evaluation, highlighting the need for architectural advances in medical dialog systems beyond simple model fine-tuning.

Abstract

Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic reasoning abilities of medical large language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating privacy and data-governance concerns. We design a robust Patient Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask" items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.

Figures (8)

• Figure 1: The landscape of MedDialogRubrics framework
• Figure 2: Multi-Agent Patient Record Generation Pipeline
• Figure 3: Patient Agent & Multi-turn Dialogue Generation Framework
• Figure 4: Generation of Multi-Agent rubrics with rejection Sampling
• Figure 5: Expert Annotation Rubric Generation Workflow