Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Combining Insights From Multiple Large Language Models Improves Diagnostic Accuracy

Gioele Barabucci, Victor Shia, Eugene Chu, Benjamin Harack, Nathan Fu

TL;DR

The use of collective intelligence methods to synthesize differential diagnoses combining the responses of different LLMs achieves two of the necessary steps towards advancing acceptance of LLMs as a diagnostic support tool: demonstrate high diagnostic accuracy and eliminate dependence on a single commercial vendor.

Abstract

Background: Large language models (LLMs) such as OpenAI's GPT-4 or Google's PaLM 2 are proposed as viable diagnostic support tools or even spoken of as replacements for "curbside consults". However, even LLMs specifically trained on medical topics may lack sufficient diagnostic accuracy for real-life applications. Methods: Using collective intelligence methods and a dataset of 200 clinical vignettes of real-life cases, we assessed and compared the accuracy of differential diagnoses obtained by asking individual commercial LLMs (OpenAI GPT-4, Google PaLM 2, Cohere Command, Meta Llama 2) against the accuracy of differential diagnoses synthesized by aggregating responses from combinations of the same LLMs. Results: We find that aggregating responses from multiple, various LLMs leads to more accurate differential diagnoses (average accuracy for 3 LLMs: $75.3\%\pm 1.6pp$) compared to the differential diagnoses produced by single LLMs (average accuracy for single LLMs: $59.0\%\pm 6.1pp$). Discussion: The use of collective intelligence methods to synthesize differential diagnoses combining the responses of different LLMs achieves two of the necessary steps towards advancing acceptance of LLMs as a diagnostic support tool: (1) demonstrate high diagnostic accuracy and (2) eliminate dependence on a single commercial vendor.

Combining Insights From Multiple Large Language Models Improves Diagnostic Accuracy

TL;DR

The use of collective intelligence methods to synthesize differential diagnoses combining the responses of different LLMs achieves two of the necessary steps towards advancing acceptance of LLMs as a diagnostic support tool: demonstrate high diagnostic accuracy and eliminate dependence on a single commercial vendor.

Abstract

Background: Large language models (LLMs) such as OpenAI's GPT-4 or Google's PaLM 2 are proposed as viable diagnostic support tools or even spoken of as replacements for "curbside consults". However, even LLMs specifically trained on medical topics may lack sufficient diagnostic accuracy for real-life applications. Methods: Using collective intelligence methods and a dataset of 200 clinical vignettes of real-life cases, we assessed and compared the accuracy of differential diagnoses obtained by asking individual commercial LLMs (OpenAI GPT-4, Google PaLM 2, Cohere Command, Meta Llama 2) against the accuracy of differential diagnoses synthesized by aggregating responses from combinations of the same LLMs. Results: We find that aggregating responses from multiple, various LLMs leads to more accurate differential diagnoses (average accuracy for 3 LLMs: ) compared to the differential diagnoses produced by single LLMs (average accuracy for single LLMs: ). Discussion: The use of collective intelligence methods to synthesize differential diagnoses combining the responses of different LLMs achieves two of the necessary steps towards advancing acceptance of LLMs as a diagnostic support tool: (1) demonstrate high diagnostic accuracy and (2) eliminate dependence on a single commercial vendor.
Paper Structure (9 sections, 2 figures, 1 table)

This paper contains 9 sections, 2 figures, 1 table.

Table of Contents

  1. Background
  2. Methods
  3. Dataset and case selection
  4. Querying of LLMs
  5. Collective intelligence and response aggregation
  6. Accuracy measure
  7. Results
  8. Discussion
  9. Acknowledgements

Figures (2)

  • Figure 1: Synthetic differential diagnoses aggregated from different LLMs show a greater diagnostic accuracy compared to differential diagnoses produced by single LLMs. This graph provides a visual representation of the data presented in Table \ref{['tab:top5-results']}. $\bigcirc$ = Cohere Command, $\triangle$ = Google PaLM 2, $\square$ = Meta Llama 2, = OpenAI GPT-4
  • Figure 2: Increasing the number of LLMs contributing to a synthetic differential leads to an increase in accuracy also (a) when the definition of correctly diagnosed case is made stricter by considering only the 3 highest ranked diagnosis in a differential (TOP-3 matching) and (b) when the top-performing LLM, GPT-4, is excluded from the experiment.