Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering

Robert Osazuwa Ness, Katie Matton, Hayden Helm, Sheng Zhang, Junaid Bajwa, Carey E. Priebe, Eric Horvitz

TL;DR

An adversarial method that is called MedFuzz, which attempts to modify benchmark questions in ways aimed at confounding the LLM, and shows promise at providing insights into the ability of an LLM to operate robustly in more realistic settings.

Abstract

Large language models (LLM) have achieved impressive performance on medical question-answering benchmarks. However, high benchmark accuracy does not imply that the performance generalizes to real-world clinical settings. Medical question-answering benchmarks rely on assumptions consistent with quantifying LLM performance but that may not hold in the open world of the clinic. Yet LLMs learn broad knowledge that can help the LLM generalize to practical conditions regardless of unrealistic assumptions in celebrated benchmarks. We seek to quantify how well LLM medical question-answering benchmark performance generalizes when benchmark assumptions are violated. Specifically, we present an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz attempts to modify benchmark questions in ways aimed at confounding the LLM. We demonstrate the approach by targeting strong assumptions about patient characteristics presented in the MedQA benchmark. Successful "attacks" modify a benchmark item in ways that would be unlikely to fool a medical expert but nonetheless "trick" the LLM into changing from a correct to an incorrect answer. Further, we present a permutation test technique that can ensure a successful attack is statistically significant. We show how to use performance on a "MedFuzzed" benchmark, as well as individual successful attacks. The methods show promise at providing insights into the ability of an LLM to operate robustly in more realistic settings.

MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering

TL;DR

An adversarial method that is called MedFuzz, which attempts to modify benchmark questions in ways aimed at confounding the LLM, and shows promise at providing insights into the ability of an LLM to operate robustly in more realistic settings.

Abstract

Large language models (LLM) have achieved impressive performance on medical question-answering benchmarks. However, high benchmark accuracy does not imply that the performance generalizes to real-world clinical settings. Medical question-answering benchmarks rely on assumptions consistent with quantifying LLM performance but that may not hold in the open world of the clinic. Yet LLMs learn broad knowledge that can help the LLM generalize to practical conditions regardless of unrealistic assumptions in celebrated benchmarks. We seek to quantify how well LLM medical question-answering benchmark performance generalizes when benchmark assumptions are violated. Specifically, we present an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz attempts to modify benchmark questions in ways aimed at confounding the LLM. We demonstrate the approach by targeting strong assumptions about patient characteristics presented in the MedQA benchmark. Successful "attacks" modify a benchmark item in ways that would be unlikely to fool a medical expert but nonetheless "trick" the LLM into changing from a correct to an incorrect answer. Further, we present a permutation test technique that can ensure a successful attack is statistically significant. We show how to use performance on a "MedFuzzed" benchmark, as well as individual successful attacks. The methods show promise at providing insights into the ability of an LLM to operate robustly in more realistic settings.
Paper Structure (32 sections, 3 figures, 2 algorithms)

This paper contains 32 sections, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. LLM performance on Medical Question-Answering
  4. Adversarially Robust Generalization
  5. Bias and fairness in medical question answering
  6. Method
  7. Workflow for applying MedFuzz
  8. Step 1: Select which benchmark assumptions to violate
  9. Step 2: Prompt attacker LLM to violate these assumptions
  10. Step 3: Evaluate overall benchmark performance after adversarial attacks
  11. Step 4: Identify interesting case studies
  12. The MedFuzz algorithm
  13. Accounting for random chance in MedFuzz Attacks
  14. Attack successes that don't generalize
  15. Ensembling multiple attacks
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Overview of the MedFuzz algorithm
  • Figure 2: Accuracy of various models on the MedQA benchmark with different numbers of MedFuzz attack attempts. The horizontal line is average human performance on USMLE exams (76.6%). GPT-4 and Claude still have human comparable performance after five attacks. BioMistral-7B is surprisingly robust to attacks. The diminishing declines in accuracy as the number of attacks increase gives insight into robustness of benchmark performance in the face of this assumption violation.
  • Figure 3: Rate of unfaithful CoT responses (i.e., those that omit references to the fuzzed information when the answer changed).