Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Q&A of Clinical Documents with Large Language Models

Ran Elgedawy, Ioana Danciu, Maria Mahbub, Sudarshan Srinivasan

TL;DR

The paper tackles the challenge of rapidly extracting key information from unstructured clinical notes in electronic health records by framing a dynamic, natural-language question-answering interface built on retrieval augmented generation and embedding models. It systematically evaluates a range of embedding-model and language-model pairings within a LangChain-based RAG pipeline on the MIMIC-IV notes corpus, identifying Wizard Vicuna 13B as the top performer despite substantial compute demands. Post-training quantization drastically reduces latency while preserving accuracy, and domain-specific fine-tuning underperforms relative to the RAG approach, highlighting the practicality of retrieval-based methods for medical QA. The results demonstrate the potential of AI-assisted clinical note querying to enhance information access in care and research, while also underscoring the need for broader datasets and robust evaluation to address model hallucinations and generalizability.

Abstract

Electronic health records (EHRs) house crucial patient data in clinical notes. As these notes grow in volume and complexity, manual extraction becomes challenging. This work introduces a natural language interface using large language models (LLMs) for dynamic question-answering on clinical notes. Our chatbot, powered by Langchain and transformer-based LLMs, allows users to query in natural language, receiving relevant answers from clinical notes. Experiments, utilizing various embedding models and advanced LLMs, show Wizard Vicuna's superior accuracy, albeit with high compute demands. Model optimization, including weight quantization, improves latency by approximately 48 times. Promising results indicate potential, yet challenges such as model hallucinations and limited diverse medical case evaluations remain. Addressing these gaps is crucial for unlocking the value in clinical notes and advancing AI-driven clinical decision-making.

Dynamic Q&A of Clinical Documents with Large Language Models

TL;DR

The paper tackles the challenge of rapidly extracting key information from unstructured clinical notes in electronic health records by framing a dynamic, natural-language question-answering interface built on retrieval augmented generation and embedding models. It systematically evaluates a range of embedding-model and language-model pairings within a LangChain-based RAG pipeline on the MIMIC-IV notes corpus, identifying Wizard Vicuna 13B as the top performer despite substantial compute demands. Post-training quantization drastically reduces latency while preserving accuracy, and domain-specific fine-tuning underperforms relative to the RAG approach, highlighting the practicality of retrieval-based methods for medical QA. The results demonstrate the potential of AI-assisted clinical note querying to enhance information access in care and research, while also underscoring the need for broader datasets and robust evaluation to address model hallucinations and generalizability.

Abstract

Electronic health records (EHRs) house crucial patient data in clinical notes. As these notes grow in volume and complexity, manual extraction becomes challenging. This work introduces a natural language interface using large language models (LLMs) for dynamic question-answering on clinical notes. Our chatbot, powered by Langchain and transformer-based LLMs, allows users to query in natural language, receiving relevant answers from clinical notes. Experiments, utilizing various embedding models and advanced LLMs, show Wizard Vicuna's superior accuracy, albeit with high compute demands. Model optimization, including weight quantization, improves latency by approximately 48 times. Promising results indicate potential, yet challenges such as model hallucinations and limited diverse medical case evaluations remain. Addressing these gaps is crucial for unlocking the value in clinical notes and advancing AI-driven clinical decision-making.
Paper Structure (21 sections, 2 figures, 7 tables)

This paper contains 21 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs in information retrieval
  4. Medical question answering using LLMs
  5. Interplay Between Embedding Models and Language Models
  6. Retrieval Augmented Generation (RAG) in Various Contexts
  7. Approach
  8. Methodology
  9. Constraints
  10. Dataset
  11. Data Preprocessing
  12. Evaluation
  13. Model Pairing Accuracy
  14. Experimental Setup
  15. Results
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Information retrieval pipeline
  • Figure 2: Accuracy of Embedding vs Language models