Demo: Statistically Significant Results On Biases and Errors of LLMs Do Not Guarantee Generalizable Results

TL;DR

The paper investigates how bias, hallucination, and omissions in medical LLMs affect patient-facing advice. It introduces an end-to-end infrastructure to generate demographically controlled, layperson-oriented prompts and to evaluate responses using multiple LLM-based evaluators, including agentic workflows. With a large-scale dataset of 3.2M prompts, 29K answers, and 684K evaluations across diverse conditions, it reports weak inter-LLM agreement (average Cohen's κ ≈ 0.118) and demonstrates that conclusions can vary with the choice of answering and evaluating LLMs. The authors advocate using multiple evaluators and reporting inter-LLM agreement to avoid overconfident, non-generalizable findings, and they release code and data to support broader bias analysis in medical chatbots.

Abstract

Recent research has shown that hallucinations, omissions, and biases are prevalent in everyday use-cases of LLMs. However, chatbots used in medical contexts must provide consistent advice in situations where non-medical factors are involved, such as when demographic information is present. In order to understand the conditions under which medical chatbots fail to perform as expected, we develop an infrastructure that 1) automatically generates queries to probe LLMs and 2) evaluates answers to these queries using multiple LLM-as-a-judge setups and prompts. For 1), our prompt creation pipeline samples the space of patient demographics, histories, disorders, and writing styles to create realistic questions that we subsequently use to prompt LLMs. In 2), our evaluation pipeline provides hallucination and omission detection using LLM-as-a-judge as well as agentic workflows, in addition to LLM-as-a-judge treatment category detectors. As a baseline study, we perform two case studies on inter-LLM agreement and the impact of varying the answering and evaluation LLMs. We find that LLM annotators exhibit low agreement scores (average Cohen's Kappa $κ=0.118$), and only specific (answering, evaluation) LLM pairs yield statistically significant differences across writing styles, genders, and races. We recommend that studies using LLM evaluation use multiple LLMs as evaluators in order to avoid arriving at statistically significant but non-generalizable results, particularly in the absence of ground-truth data. We also suggest publishing inter-LLM agreement metrics for transparency. Our code and dataset are available here: https://github.com/BBN-E/medic-neurips-2025-demo.

Figures (12)

• Figure 1: Our pipeline for generating diverse prompts. First, a Patient profile is generated using random medical history and demographic data. These are randomly included/excluded, resulting in a Patient Expression. A Desire and the Patient info are combined to generate a Question, which is subsequently combined with the Patient Expression into the final LLM prompt. An optional Style is used to restyle parts of the prompt.
• Figure 2: Manage, Visit, and Resource average rates for each LLM. 95% confidence intervals are shown.
• Figure 3: Average detection count of Hallucinations and Omissions for each evaluator LLM. 95% confidence intervals are shown.
• Figure 4: Manage, Visit, and Resource average rates in our dataset based on style. Each column of plots corresponds to a different evaluator LLM whereas each row corresponds to a different answering LLM. 95% confidence intervals are shown.
• Figure 5: Manage, Visit, and Resource average rates in our dataset based on gender. The x-axis of subplots varies the evaluator LLM whereas the y-axis varies the answering LLM. 95% confidence intervals are shown.