Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations

Yahan Li, Xinyi Jie, Wanjia Ruan, Xubei Zhang, Huaijie Zhu, Yicheng Gao, Chaohao Du, Ruishan Liu

Abstract

Large language models (LLMs) are increasingly used for medical consultation and health information support. In this high-stakes setting, safety depends not only on medical knowledge, but also on how models respond when patient inputs are unclear, inconsistent, or misleading. However, most existing medical LLM evaluations assume idealized and well-posed patient questions, which limits their realism. In this paper, we study challenging patient behaviors that commonly arise in real medical consultations and complicate safe clinical reasoning. We define four clinically grounded categories of such behaviors: information contradiction, factual inaccuracy, self-diagnosis, and care resistance. For each behavior, we specify concrete failure criteria that capture unsafe responses. Building on four existing medical dialogue datasets, we introduce CPB-Bench (Challenging Patient Behaviors Benchmark), a bilingual (English and Chinese) benchmark of 692 multi-turn dialogues annotated with these behaviors. We evaluate a range of open- and closed-source LLMs on their responses to challenging patient utterances. While models perform well overall, we identify consistent, behavior-specific failure patterns, with particular difficulty in handling contradictory or medically implausible patient information. We also study four intervention strategies and find that they yield inconsistent improvements and can introduce unnecessary corrections. We release the dataset and code.

Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations

Abstract

Large language models (LLMs) are increasingly used for medical consultation and health information support. In this high-stakes setting, safety depends not only on medical knowledge, but also on how models respond when patient inputs are unclear, inconsistent, or misleading. However, most existing medical LLM evaluations assume idealized and well-posed patient questions, which limits their realism. In this paper, we study challenging patient behaviors that commonly arise in real medical consultations and complicate safe clinical reasoning. We define four clinically grounded categories of such behaviors: information contradiction, factual inaccuracy, self-diagnosis, and care resistance. For each behavior, we specify concrete failure criteria that capture unsafe responses. Building on four existing medical dialogue datasets, we introduce CPB-Bench (Challenging Patient Behaviors Benchmark), a bilingual (English and Chinese) benchmark of 692 multi-turn dialogues annotated with these behaviors. We evaluate a range of open- and closed-source LLMs on their responses to challenging patient utterances. While models perform well overall, we identify consistent, behavior-specific failure patterns, with particular difficulty in handling contradictory or medically implausible patient information. We also study four intervention strategies and find that they yield inconsistent improvements and can introduce unnecessary corrections. We release the dataset and code.

Paper Structure

This paper contains 63 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLMs for Health Information Support and Medical Dialogue
  4. Safety Risks in Medical Settings
  5. Adversarial Testing of LLMs and Medical LLMs
  6. Patient Communication Challenges and Gaps in Current Evaluation
  7. Challenging Patient Behaviors in Medical Consultations
  8. Challenging Patient Behavior Taxonomy
  9. 1. Information Contradiction.
  10. 2. Factual Inaccuracy.
  11. 3. Self-diagnosis.
  12. 4. Care Resistance.
  13. Datasets and Annotation
  14. Datasets.
  15. Annotation Protocol.
  16. ...and 48 more sections

Figures (4)

  • Figure 1: Overview of our evaluation pipeline. We define four behavior categories, evaluate failures in first responses, and use multi-turn follow-ups for failed cases, alongside interventions and controlled synthetic cases.
  • Figure 2: Evaluation under challenging patient behaviors. Left: failure counts by type across models (GPT-4o-filtered, human-verified). Right: number of failures corrected in multi-turn follow-up relative to total failures.
  • Figure 3: Distribution of positive-case turn positions across datasets, measured as the normalized turn index (turn_index / total_turns). Positive cases tend to occur throughout the dialogue, with a slight concentration in later turns.
  • Figure 4: Distribution of sampled truncation points for negative cases across datasets. Truncation positions are bootstrapped from the positive-case distribution to match its turn-position profile, ensuring comparable interactional conditions.