Rx Strategist: Prescription Verification using LLM Agents System

TL;DR

This work offers a new approach - Rx Strategist - that makes use of knowledge graphs and different search strategies to enhance the power of Large Language Models inside an agentic framework, allowing for a multi-stage LLM pipeline and reliable information retrieval from a custom-built active ingredient database.

Abstract

To protect patient safety, modern pharmaceutical complexity demands strict prescription verification. We offer a new approach - Rx Strategist - that makes use of knowledge graphs and different search strategies to enhance the power of Large Language Models (LLMs) inside an agentic framework. This multifaceted technique allows for a multi-stage LLM pipeline and reliable information retrieval from a custom-built active ingredient database. Different facets of prescription verification, such as indication, dose, and possible drug interactions, are covered in each stage of the pipeline. We alleviate the drawbacks of monolithic LLM techniques by spreading reasoning over these stages, improving correctness and reliability while reducing memory demands. Our findings demonstrate that Rx Strategist surpasses many current LLMs, achieving performance comparable to that of a highly experienced clinical pharmacist. In the complicated world of modern medications, this combination of LLMs with organized knowledge and sophisticated search methods presents a viable avenue for reducing prescription errors and enhancing patient outcomes.

Figures (7)

• Figure 1: Benchmark of Drug Verification on our system against various levels of Clinical Pharmacists' Evaluation. The result shows that our system can perform at the level of senior pharmacists with five years of experience (YoE), while state-of-the-art LLMs (Llama 3.1 70B) fall short of even junior pharmacists' performance.
• Figure 2: An overview of Rx Strategist. The process begins with extracting key information from the prescription, including the diagnosis, prescribed dosage, and active ingredients. This information is then passed to the indication verification module, which first identifies the ICD-10 code associated with the indicated condition and then cross-references it with the patient's diagnosis to ascertain whether the prescribed active ingredients are appropriate for treatment. Following this verification, the relevant active ingredients proceed to the dosage retriever module, which assesses whether the prescribed dosage falls within the recommended range for the patient's specific characteristics. Finally, the checker module consolidates the information from both the indication verification and dosage retriever stages, providing a comprehensive assessment and conclusion regarding the appropriateness of the prescription.
• Figure 3: Some visualizations on the statistics of the active ingredient dataset. The left chart shows the discrepancy in adult and pediatric in quantity. The center chart displays the distribution in the number of words described in each drug. The right one indicates how many diseases each particular medicine can cure. AIs = Active Ingredients.
• Figure 4: The relationship between the precision-recall ratio and the F0.5 score in prescription verification tasks. The prediction that strike a balance between precision and recall, thereby minimizing both false positives and false negatives, generally achieve higher F0.5 scores.
• Figure 5: This is a sample of the element Afentanil and its related interaction. The center element branches to other elements connected by their corresponding relationship. The captured relationships are elements of interaction with Afentanil, the eligible age for usage such as Adults, and its adverse effects such as Hypotension.