Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Replication of LLM Mistakes in Medical Conversations

Oleksii Proniakin, Diego Fajardo, Ruslan Nazarenko, Razvan Marinescu

TL;DR

This work tackles the gap between holistic evaluation of clinical LLM conversations and the need for per-turn improvement signals. It introduces MedMistake, an automated pipeline that generates LLM-driven patient–doctor dialogues, uses a two-LLM judge committee to pinpoint mistakes across dozens of dimensions, and converts these mistakes into single-shot QA benchmarks (MedMistake-Bench and MedMistake-All). The approach yields thousands of mistakes, validates a substantial subset with medical experts, and benchmarks 12 frontier LLMs, revealing varying strengths and the value of automatic mistake synthesis for scalable model refinement. The method is largely automated, scalable across domains, and provides a practical pathway for robust, per-turn improvement in medical AI systems.

Abstract

Large language models (LLMs) are increasingly evaluated in clinical settings using multi-dimensional rubrics which quantify reasoning quality, safety, and patient-centeredness. Yet, replicating specific mistakes in other LLM models is not straightforward and often requires manual effort. We introduce MedMistake, an automatic pipeline that extracts mistakes LLMs make in patient-doctor conversations and converts them into a benchmark of single-shot QA pairs. Our pipeline (1) creates complex, conversational data between an LLM patient and LLM doctor, (2) runs an evaluation with a committee of 2 LLM judges across a variety of dimensions and (3) creates simplified single-shot QA scenarios from those mistakes. We release MedMistake-All, a dataset of 3,390 single-shot QA pairs where GPT-5 and Gemini 2.5 Pro are currently failing to answer correctly, as judged by two LLM judges. We used medical experts to validate a subset of 211/3390 questions (MedMistake-Bench), which we used to run a final evaluation of 12 frontier LLMs: Claude Opus 4.5, Claude Sonnet 4.5, DeepSeek-Chat, Gemini 2.5 Pro, Gemini 3 Pro, GPT-4o, GPT-5, GPT-5.1, GPT-5.2, Grok 4, Grok 4.1, Mistral Large. We found that GPT models, Claude and Grok obtained the best performance on MedMistake-Bench. We release both the doctor-validated benchmark (MedMistake-Bench), as well as the full dataset (MedMistake-All) at https://huggingface.co/datasets/TheLumos/MedicalMistakeBenchmark.

Automatic Replication of LLM Mistakes in Medical Conversations

TL;DR

This work tackles the gap between holistic evaluation of clinical LLM conversations and the need for per-turn improvement signals. It introduces MedMistake, an automated pipeline that generates LLM-driven patient–doctor dialogues, uses a two-LLM judge committee to pinpoint mistakes across dozens of dimensions, and converts these mistakes into single-shot QA benchmarks (MedMistake-Bench and MedMistake-All). The approach yields thousands of mistakes, validates a substantial subset with medical experts, and benchmarks 12 frontier LLMs, revealing varying strengths and the value of automatic mistake synthesis for scalable model refinement. The method is largely automated, scalable across domains, and provides a practical pathway for robust, per-turn improvement in medical AI systems.

Abstract

Large language models (LLMs) are increasingly evaluated in clinical settings using multi-dimensional rubrics which quantify reasoning quality, safety, and patient-centeredness. Yet, replicating specific mistakes in other LLM models is not straightforward and often requires manual effort. We introduce MedMistake, an automatic pipeline that extracts mistakes LLMs make in patient-doctor conversations and converts them into a benchmark of single-shot QA pairs. Our pipeline (1) creates complex, conversational data between an LLM patient and LLM doctor, (2) runs an evaluation with a committee of 2 LLM judges across a variety of dimensions and (3) creates simplified single-shot QA scenarios from those mistakes. We release MedMistake-All, a dataset of 3,390 single-shot QA pairs where GPT-5 and Gemini 2.5 Pro are currently failing to answer correctly, as judged by two LLM judges. We used medical experts to validate a subset of 211/3390 questions (MedMistake-Bench), which we used to run a final evaluation of 12 frontier LLMs: Claude Opus 4.5, Claude Sonnet 4.5, DeepSeek-Chat, Gemini 2.5 Pro, Gemini 3 Pro, GPT-4o, GPT-5, GPT-5.1, GPT-5.2, Grok 4, Grok 4.1, Mistral Large. We found that GPT models, Claude and Grok obtained the best performance on MedMistake-Bench. We release both the doctor-validated benchmark (MedMistake-Bench), as well as the full dataset (MedMistake-All) at https://huggingface.co/datasets/TheLumos/MedicalMistakeBenchmark.
Paper Structure (18 sections, 3 figures, 2 tables)

This paper contains 18 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work:
  3. Methods
  4. MedMistake-Bench Pipeline
  5. Results
  6. Discussion
  7. Limitations
  8. Future Work
  9. Prompts Used
  10. Prompts Used
  11. Score Mistake Prompt (Quality Evaluation)
  12. Scenario Generation Prompt
  13. Score Scenario Prompt (Quality Evaluation)
  14. Judge Answer Prompt (Used in Scenario Testing)
  15. Replication Testing and Scoring Prompt (Boolean Judge)
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Overview of the MedMistake-Bench pipeline. We first synthesize conversations using MedPIfajardo2025medpi (blue), then extract mistakes from those conversations which we distill into single-shot QA pairs (green), and finally we run a medical expert validation (yellow).
  • Figure 2: Example snippet from a generated medical conversation between an LLM patient and LLM doctor, where the LLM doctor makes a mistake in a drug prescription. The mistake is identified by a medical committee of 2 LLM judges, and a single-shot clinical scenario is generated that is used to score LLMs.
  • Figure 3: Distribution of mistakes that we considered, showing the proportion of mistakes reproduced by either Gemini 2.5 Pro or GPT-5.