Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning

Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, Yulia Tsvetkov

TL;DR

This paper proposes to change the static paradigm to an interactive one, develop systems that proactively ask questions to gather more information and respond reliably, and introduces an interactive MediQ benchmark and a novel question-asking system to evaluate question-asking ability in LLMs.

Abstract

Users typically engage with LLMs interactively, yet most existing benchmarks evaluate them in a static, single-turn format, posing reliability concerns in interactive scenarios. We identify a key obstacle towards reliability: LLMs are trained to answer any question, even with incomplete context or insufficient knowledge. In this paper, we propose to change the static paradigm to an interactive one, develop systems that proactively ask questions to gather more information and respond reliably, and introduce an benchmark - MediQ - to evaluate question-asking ability in LLMs. MediQ simulates clinical interactions consisting of a Patient System and an adaptive Expert System; with potentially incomplete initial information, the Expert refrains from making diagnostic decisions when unconfident, and instead elicits missing details via follow-up questions. We provide a pipeline to convert single-turn medical benchmarks into an interactive format. Our results show that directly prompting state-of-the-art LLMs to ask questions degrades performance, indicating that adapting LLMs to proactive information-seeking settings is nontrivial. We experiment with abstention strategies to better estimate model confidence and decide when to ask questions, improving diagnostic accuracy by 22.3%; however, performance still lags compared to an (unrealistic in practice) upper bound with complete information upfront. Further analyses show improved interactive performance with filtering irrelevant contexts and reformatting conversations. Overall, we introduce a novel problem towards LLM reliability, an interactive MediQ benchmark and a novel question-asking system, and highlight directions to extend LLMs' information-seeking abilities in critical domains.

MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning

TL;DR

This paper proposes to change the static paradigm to an interactive one, develop systems that proactively ask questions to gather more information and respond reliably, and introduces an interactive MediQ benchmark and a novel question-asking system to evaluate question-asking ability in LLMs.

Abstract

Users typically engage with LLMs interactively, yet most existing benchmarks evaluate them in a static, single-turn format, posing reliability concerns in interactive scenarios. We identify a key obstacle towards reliability: LLMs are trained to answer any question, even with incomplete context or insufficient knowledge. In this paper, we propose to change the static paradigm to an interactive one, develop systems that proactively ask questions to gather more information and respond reliably, and introduce an benchmark - MediQ - to evaluate question-asking ability in LLMs. MediQ simulates clinical interactions consisting of a Patient System and an adaptive Expert System; with potentially incomplete initial information, the Expert refrains from making diagnostic decisions when unconfident, and instead elicits missing details via follow-up questions. We provide a pipeline to convert single-turn medical benchmarks into an interactive format. Our results show that directly prompting state-of-the-art LLMs to ask questions degrades performance, indicating that adapting LLMs to proactive information-seeking settings is nontrivial. We experiment with abstention strategies to better estimate model confidence and decide when to ask questions, improving diagnostic accuracy by 22.3%; however, performance still lags compared to an (unrealistic in practice) upper bound with complete information upfront. Further analyses show improved interactive performance with filtering irrelevant contexts and reformatting conversations. Overall, we introduce a novel problem towards LLM reliability, an interactive MediQ benchmark and a novel question-asking system, and highlight directions to extend LLMs' information-seeking abilities in critical domains.
Paper Structure (53 sections, 4 equations, 14 figures, 6 tables)

This paper contains 53 sections, 4 equations, 14 figures, 6 tables.

Table of Contents

  1. Introduction
  2. MediQ: Dynamic Medical Consultation Framework Overview
  3. The Patient System
  4. The Expert System
  5. Expert System Breakdown
  6. Expert System Variants with Different Abstention Strategies
  7. Experiments
  8. Patient System Reliability Evaluation
  9. Expert System Experiments
  10. Benchmarking Existing LLMs in Incomplete Information Scenarios
  11. Interactive Expert Systems
  12. Results
  13. How reliable is the MediQ Patient system?
  14. How do existing Non-Interactive LLMs perform with Limited Information?
  15. How much of the performance gap can be closed by asking questions?
  16. ...and 38 more sections

Figures (14)

  • Figure 1: Information Seeking Task. In standard medical QA tasks (left), all necessary information is given to the assistant model at the same time. When given partial information, current LLMs only provides general responses (middle). In a more realistic scenario (right), the presentation of patient information relies on proactive elicitation from the doctor; our proposed MediQ framework operationalizes this scenario.
  • Figure 1: Patient system reliability.
  • Figure 2: The MediQ Benchmark. MediQ operationalizes a more realistic dynamic clinical interaction between a Patient system and an Expert system to evaluate info-seeking and question-asking.
  • Figure 3: Expert system information flow breakdown.
  • Figure 4: Non-interactive Expert system evaluation at various information availability levels. The question and options are provided to the Expert model in all three settings.
  • ...and 9 more figures