Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantifying Hallucinations in Language Language Models on Medical Textbooks

Brandon C. Colelough, Davis Bartels, Dina Demner-Fushman

TL;DR

This work determines the prevalence of hallucinations for a prominent open source large language model (LLaMA-70B-Instruct) in medical QA given novel prompts, and determines clinician preference to model responses.

Abstract

Hallucinations, the tendency for large language models to provide responses with factually incorrect and unsupported claims, is a serious problem within natural language processing for which we do not yet have an effective solution to mitigate against. Existing benchmarks for medical QA rarely evaluate this behavior against a fixed evidence source. We ask how often hallucinations occur on textbook-grounded QA and how responses to medical QA prompts vary across models. We conduct two experiments: the first experiment to determine the prevalence of hallucinations for a prominent open source large language model (LLaMA-70B-Instruct) in medical QA given novel prompts, and the second experiment to determine the prevalence of hallucinations and clinician preference to model responses. We observed, in experiment one, with the passages provided, LLaMA-70B-Instruct hallucinated in 19.7\% of answers (95\% CI 18.6 to 20.7) even though 98.8\% of prompt responses received maximal plausibility, and observed in experiment two, across models, lower hallucination rates aligned with higher usefulness scores ($ρ=-0.71$, $p=0.058$). Clinicians produced high agreement (quadratic weighted $κ=0.92$) and ($τ_b=0.06$ to $0.18$, $κ=0.57$ to $0.61$) for experiments 1 and ,2 respectively

Quantifying Hallucinations in Language Language Models on Medical Textbooks

TL;DR

This work determines the prevalence of hallucinations for a prominent open source large language model (LLaMA-70B-Instruct) in medical QA given novel prompts, and determines clinician preference to model responses.

Abstract

Hallucinations, the tendency for large language models to provide responses with factually incorrect and unsupported claims, is a serious problem within natural language processing for which we do not yet have an effective solution to mitigate against. Existing benchmarks for medical QA rarely evaluate this behavior against a fixed evidence source. We ask how often hallucinations occur on textbook-grounded QA and how responses to medical QA prompts vary across models. We conduct two experiments: the first experiment to determine the prevalence of hallucinations for a prominent open source large language model (LLaMA-70B-Instruct) in medical QA given novel prompts, and the second experiment to determine the prevalence of hallucinations and clinician preference to model responses. We observed, in experiment one, with the passages provided, LLaMA-70B-Instruct hallucinated in 19.7\% of answers (95\% CI 18.6 to 20.7) even though 98.8\% of prompt responses received maximal plausibility, and observed in experiment two, across models, lower hallucination rates aligned with higher usefulness scores (, ). Clinicians produced high agreement (quadratic weighted ) and ( to , to ) for experiments 1 and ,2 respectively
Paper Structure (34 sections, 5 equations, 4 figures, 3 tables)

This paper contains 34 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Research Focus
  5. Research Questions
  6. Corpus and Prompt Construction
  7. Experiment 1: Baseline Hallucination Detection
  8. Experiment 2: Cross-Model Evaluation and Preference Study
  9. Metrics and Statistical Analysis
  10. Reproducibility
  11. Pipeline and environment.
  12. Human-in-the-loop portals.
  13. Data releases.
  14. Evaluation Protocol
  15. Plausibility & Answerability
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Model–QA type heat-map. Each cell shows the mean annotator rank (1 = best, 8 = worst)
  • Figure 2: Mean Likert score per model and QA type. Each cell shows the weighted mean Likert score (0 = Bad, 2 = Good).
  • Figure 3: Annotator bias matrix (weighted). Colour shows the weighted mean rank of each model for each annotator, where weights are the number of Judgements that the annotator supplied. A black rectangular outline highlights an annotators favourite model.
  • Figure 4: Weighted Likert bias matrix. Colour shows the weighted mean Likert score for each (annotator, model) cell, where weights equal the number of judgements that annotator provided for that model.

Theorems & Definitions (1)

  • Definition 1