Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Clean & Clear: Feasibility of Safe LLM Clinical Guidance

Julia Ive, Felix Jozsa, Nick Jackson, Paulina Bondaronek, Ciaran Scott Hill, Richard Dobson

TL;DR

Clinicians rely on local guidelines for safe care, but large language models can hallucinate, risking patient safety. The authors implement a safety-first approach by constraining a locally deployed Llama-3.1-8B to extract and return verbatim guideline lines from UCLH documents, rather than generating new content. In a pilot with seven clinicians and six guidelines, the system achieved perfect recall (no missed critical information) and substantial relevance with notable time savings (roughly one-third the manual retrieval time), while exposing areas for improvement in question formulation and broader deployment. The study supports the feasibility of hospital-ready AI-assisted guideline access and outlines concrete steps for broader evaluation, integration with health IT systems, and expansion to more guidelines to enhance patient safety and clinician efficiency.

Abstract

Background: Clinical guidelines are central to safe evidence-based medicine in modern healthcare, providing diagnostic criteria, treatment options and monitoring advice for a wide range of illnesses. LLM-empowered chatbots have shown great promise in Healthcare Q&A tasks, offering the potential to provide quick and accurate responses to medical inquiries. Our main objective was the development and preliminary assessment of an LLM-empowered chatbot software capable of reliably answering clinical guideline questions using University College London Hospital (UCLH) clinical guidelines. Methods: We used the open-weight Llama-3.1-8B LLM to extract relevant information from the UCLH guidelines to answer questions. Our approach highlights the safety and reliability of referencing information over its interpretation and response generation. Seven doctors from the ward assessed the chatbot's performance by comparing its answers to the gold standard. Results: Our chatbot demonstrates promising performance in terms of relevance, with ~73% of its responses rated as very relevant, showcasing a strong understanding of the clinical context. Importantly, our chatbot achieves a recall of 1.00 for extracted guideline lines, substantially minimising the risk of missing critical information. Approximately 78% of responses were rated satisfactory in terms of completeness. A small portion (~14.5%) contained minor unnecessary information, indicating occasional lapses in precision. The chatbot' showed high efficiency, with an average completion time of 10 seconds, compared to 30 seconds for human respondents. Evaluation of clinical reasoning showed that 72% of the chatbot's responses were without flaws. Our chatbot demonstrates significant potential to speed up and improve the process of accessing locally relevant clinical information for healthcare professionals.

Clean & Clear: Feasibility of Safe LLM Clinical Guidance

TL;DR

Clinicians rely on local guidelines for safe care, but large language models can hallucinate, risking patient safety. The authors implement a safety-first approach by constraining a locally deployed Llama-3.1-8B to extract and return verbatim guideline lines from UCLH documents, rather than generating new content. In a pilot with seven clinicians and six guidelines, the system achieved perfect recall (no missed critical information) and substantial relevance with notable time savings (roughly one-third the manual retrieval time), while exposing areas for improvement in question formulation and broader deployment. The study supports the feasibility of hospital-ready AI-assisted guideline access and outlines concrete steps for broader evaluation, integration with health IT systems, and expansion to more guidelines to enhance patient safety and clinician efficiency.

Abstract

Background: Clinical guidelines are central to safe evidence-based medicine in modern healthcare, providing diagnostic criteria, treatment options and monitoring advice for a wide range of illnesses. LLM-empowered chatbots have shown great promise in Healthcare Q&A tasks, offering the potential to provide quick and accurate responses to medical inquiries. Our main objective was the development and preliminary assessment of an LLM-empowered chatbot software capable of reliably answering clinical guideline questions using University College London Hospital (UCLH) clinical guidelines. Methods: We used the open-weight Llama-3.1-8B LLM to extract relevant information from the UCLH guidelines to answer questions. Our approach highlights the safety and reliability of referencing information over its interpretation and response generation. Seven doctors from the ward assessed the chatbot's performance by comparing its answers to the gold standard. Results: Our chatbot demonstrates promising performance in terms of relevance, with ~73% of its responses rated as very relevant, showcasing a strong understanding of the clinical context. Importantly, our chatbot achieves a recall of 1.00 for extracted guideline lines, substantially minimising the risk of missing critical information. Approximately 78% of responses were rated satisfactory in terms of completeness. A small portion (~14.5%) contained minor unnecessary information, indicating occasional lapses in precision. The chatbot' showed high efficiency, with an average completion time of 10 seconds, compared to 30 seconds for human respondents. Evaluation of clinical reasoning showed that 72% of the chatbot's responses were without flaws. Our chatbot demonstrates significant potential to speed up and improve the process of accessing locally relevant clinical information for healthcare professionals.

Paper Structure

This paper contains 17 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Study Aims and Objectives
  3. Methods
  4. Ethical Considerations
  5. LLM Adaptation
  6. Evaluation Design
  7. Participants
  8. Tasks
  9. Data
  10. Results
  11. Automatic Evaluation
  12. Human Evaluation
  13. Discussion
  14. Statement of key findings
  15. Strengths and Limitations
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Our Chatbot Providing Investigation Recommendations for Hypokalaemia. This screenshot displays an interaction with our chatbot.
  • Figure 2: Task Relevance and Completeness Distribution Across Tasks. Relevance distribution (left chart) shows that the majority of responses were deemed "Very relevant" (84.6%). Completeness distribution (right chart) categorises responses based on the presence of unnecessary or missing information. A substantial majority (82.1%) of responses were marked as "Satisfactory".
  • Figure 3: Task Relevance and Completeness Distribution Across Scenarios. Relevance distribution (left chart) shows that the majority of responses were deemed "Relevant" (50.0%). Completeness Distribution (right chart) categorises responses based on the presence of unnecessary or missing information. A substantial portion (68.8%) of responses were marked as "Satisfactory".