CPGPrompt: Translating Clinical Guidelines into LLM-Executable Decision Support

TL;DR

CPGPrompt presents a generalizable framework that auto-translates narrative clinical practice guidelines into auditable, executable decision trees and a chatbot-based execution engine. By evaluating across three domains with synthetic vignettes, it demonstrates high recall for binary referral decisions and variable but domain-influenced performance for fine-grained pathway classification, highlighting strengths in safety and transparency while outlining limitations in negation handling and temporal reasoning. The approach offers a practical path toward domain-agnostic guideline-based decision support with auditable reasoning, potentially improving trust, regulatory readiness, and cross-domain applicability. Future work should enhance temporal and uncertainty reasoning and explore hybrid human-AI workflows to address remaining clinical judgment gaps.

Abstract

Clinical practice guidelines (CPGs) provide evidence-based recommendations for patient care; however, integrating them into Artificial Intelligence (AI) remains challenging. Previous approaches, such as rule-based systems, face significant limitations, including poor interpretability, inconsistent adherence to guidelines, and narrow domain applicability. To address this, we develop and validate CPGPrompt, an auto-prompting system that converts narrative clinical guidelines into large language models (LLMs). Our framework translates CPGs into structured decision trees and utilizes an LLM to dynamically navigate them for patient case evaluation. Synthetic vignettes were generated across three domains (headache, lower back pain, and prostate cancer) and distributed into four categories to test different decision scenarios. System performance was assessed on both binary specialty-referral decisions and fine-grained pathway-classification tasks. The binary specialty referral classification achieved consistently strong performance across all domains (F1: 0.85-1.00), with high recall (1.00 $\pm$ 0.00). In contrast, multi-class pathway assignment showed reduced performance, with domain-specific variations: headache (F1: 0.47), lower back pain (F1: 0.72), and prostate cancer (F1: 0.77). Domain-specific performance differences reflected the structure of each guideline. The headache guideline highlighted challenges with negation handling. The lower back pain guideline required temporal reasoning. In contrast, prostate cancer pathways benefited from quantifiable laboratory tests, resulting in more reliable decision-making.

Figures (4)

• Figure 1: Overview of CPGPrompt workflow. Clinical practice guidelines are first parsed into structured guidance trees, which are then combined with patient cases and traversed using LLM-based queries to produce referral or management actions.
• Figure 2: Example prompt and output illustrating how CPGPrompt converts CPG into a structured JSON-based guidance tree, demonstrating both simple feature check nodes and multi-criteria check nodes.
• Figure 3: Example of generating a single-criteria vignette in the headache domain. The figure shows the prompt template provided to the LLM (left) and a sample vignette output describing a patient with thunderclap headache (right).
• Figure 4: Distribution of step differences by optimal step group. The x-axis groups cases by the number of steps required in the guideline-defined optimal path, while the y-axis shows the difference between predicted and optimal traversal lengths (predicted -- optimal). Negative values indicate early termination. Violin plots represent the distribution of differences, with horizontal bars showing the mean. The red dashed line at 0 marks perfect efficiency.