Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How well do LLMs cite relevant medical references? An evaluation framework and analyses

Kevin Wu, Eric Wu, Ally Cassasola, Angela Zhang, Kevin Wei, Teresa Nguyen, Sith Riantawan, Patricia Shi Riantawan, Daniel E. Ho, James Zou

TL;DR

This study introduces SourceCheckup, an automated pipeline to evaluate whether medical LLM outputs are properly supported by cited sources. It demonstrates that GPT-4 is a strong, doctor-aligned evaluator of source attribution and benchmarks five leading LLMs on 1200 questions and over 40K statement-source pairs, revealing that a large fraction of responses lack full source support, even with retrieval augmentation. The authors provide a curated dataset and expert validation to enable ongoing benchmarking and highlight the need for explicit source verification in clinical AI. The work has regulatory and practical implications, underscoring that trustworthy, auditable medical references are essential for safe deployment of LLMs in healthcare.

Abstract

Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called \textit{SourceCheckup} and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30\% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.

How well do LLMs cite relevant medical references? An evaluation framework and analyses

TL;DR

This study introduces SourceCheckup, an automated pipeline to evaluate whether medical LLM outputs are properly supported by cited sources. It demonstrates that GPT-4 is a strong, doctor-aligned evaluator of source attribution and benchmarks five leading LLMs on 1200 questions and over 40K statement-source pairs, revealing that a large fraction of responses lack full source support, even with retrieval augmentation. The authors provide a curated dataset and expert validation to enable ongoing benchmarking and highlight the need for explicit source verification in clinical AI. The work has regulatory and practical implications, underscoring that trustworthy, auditable medical references are essential for safe deployment of LLMs in healthcare.

Abstract

Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called \textit{SourceCheckup} and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30\% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.
Paper Structure (24 sections, 4 equations, 9 figures, 8 tables)

This paper contains 24 sections, 4 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. LLM Evaluations
  5. SourceCheckup Evaluation Framework
  6. Question Generation
  7. LLM Question Answering
  8. Statement Parsing
  9. URL Source Parsing
  10. Source Verification
  11. Expert validation of GPT-4 Automated Tasks
  12. Question Generation and Response Parser
  13. Source Verification
  14. URL Analysis
  15. Results
  16. ...and 9 more sections

Figures (9)

  • Figure 1: Evaluation of the quality of source verification in LLMs on medical queries. Each model is evaluated on three metrics. Source URL Validity measures the proportion of generated URLs that return a valid webpage. Statement-level Support measures the percentage of statements that are supported by at least one source in the same response. Response-level Support measures the percentage of responses that have all their statements supported. Full numerical results are displayed in Table \ref{['tab:main_results']}.
  • Figure 2: Schematic of the SourceCheckup evaluation pipeline. To start, GPT-4 generates a question based on a given medical reference text. Each evaluated LLM produces a response based on this question, which includes the response text along with any URL sources. The LLM response is parsed for individual medical statements, while the URL sources are downloaded. Finally, the Source Verification model is asked to determine whether a given medical statement is supported by the source text and to provide a reason for the decision.
  • Figure 3: An end-to-end example of the SourceCheckup evaluation framework based on a real response from GPT-4 (RAG). A question is generated based on the contents of a medical reference text. The question is posed to an LLM, and the response is parsed into statements and sources. Each statement-source pair is automatically scored by the Source Verification model as supported (i.e. the source contains evidence to support the statement) or not supported.
  • Figure 4: Agreement between the Source Verification model and doctors on the task of source verification. We asked three medical doctors (D1, D2, and D3) to determine whether pairs of statements and source texts are supported or unsupported. We found that the Source Verification model has a higher agreement with the doctor consensus than the average agreement between doctors.
  • Figure 5: Statements produced by GPT-4 (RAG) found to be unsupported by the Source Verification model. In the first example, the source provides information contrary to the statement in the response. In the second example, the statement is unsupported since it cannot be substantiated in the provided source.
  • ...and 4 more figures