Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models for Drug Overdose Prediction from Longitudinal Medical Records

Md Sultan Al Nahian, Chris Delcher, Daniel Harris, Peter Akpunonu, Ramakanth Kavuluru

TL;DR

This study evaluates GPT-4o for predicting drug overdose events from longitudinal insurance claims in the Merative MarketScan dataset, framing overdose risk as a sequential prediction task across $7$- and $30$-day windows. It contrasts traditional baselines (Random Forest, XGBoost) with multiple LLM input representations, comparing zero-shot and fine-tuned GPT-4o performance. Fine-tuning on aggregated visit-level statistics yields the strongest results, with an $F1$ score of $84.53$ (recall $82$) for the $7$-day window, outperforming baselines, while zero-shot prompts demonstrate meaningful performance leveraging LLM prior knowledge. The work highlights input design, code-vs-description formats, and context length as key factors for LLM-based clinical prediction, and discusses cost and deployment considerations for integrating such models into decision-support pipelines.

Abstract

The ability to predict drug overdose risk from a patient's medical records is crucial for timely intervention and prevention. Traditional machine learning models have shown promise in analyzing longitudinal medical records for this task. However, recent advancements in large language models (LLMs) offer an opportunity to enhance prediction performance by leveraging their ability to process long textual data and their inherent prior knowledge across diverse tasks. In this study, we assess the effectiveness of Open AI's GPT-4o LLM in predicting drug overdose events using patients' longitudinal insurance claims records. We evaluate its performance in both fine-tuned and zero-shot settings, comparing them to strong traditional machine learning methods as baselines. Our results show that LLMs not only outperform traditional models in certain settings but can also predict overdose risk in a zero-shot setting without task-specific training. These findings highlight the potential of LLMs in clinical decision support, particularly for drug overdose risk prediction.

Large Language Models for Drug Overdose Prediction from Longitudinal Medical Records

TL;DR

This study evaluates GPT-4o for predicting drug overdose events from longitudinal insurance claims in the Merative MarketScan dataset, framing overdose risk as a sequential prediction task across - and -day windows. It contrasts traditional baselines (Random Forest, XGBoost) with multiple LLM input representations, comparing zero-shot and fine-tuned GPT-4o performance. Fine-tuning on aggregated visit-level statistics yields the strongest results, with an score of (recall ) for the -day window, outperforming baselines, while zero-shot prompts demonstrate meaningful performance leveraging LLM prior knowledge. The work highlights input design, code-vs-description formats, and context length as key factors for LLM-based clinical prediction, and discusses cost and deployment considerations for integrating such models into decision-support pipelines.

Abstract

The ability to predict drug overdose risk from a patient's medical records is crucial for timely intervention and prevention. Traditional machine learning models have shown promise in analyzing longitudinal medical records for this task. However, recent advancements in large language models (LLMs) offer an opportunity to enhance prediction performance by leveraging their ability to process long textual data and their inherent prior knowledge across diverse tasks. In this study, we assess the effectiveness of Open AI's GPT-4o LLM in predicting drug overdose events using patients' longitudinal insurance claims records. We evaluate its performance in both fine-tuned and zero-shot settings, comparing them to strong traditional machine learning methods as baselines. Our results show that LLMs not only outperform traditional models in certain settings but can also predict overdose risk in a zero-shot setting without task-specific training. These findings highlight the potential of LLMs in clinical decision support, particularly for drug overdose risk prediction.

Paper Structure

This paper contains 17 sections, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Datasets
  3. Cohort Selection
  4. Data Pre-processing
  5. Methods
  6. Baseline Methods
  7. Large Language Models
  8. Experimental Setup
  9. Results
  10. Discussion
  11. Number of Encounters
  12. Influence of Different Fields
  13. Descriptions Vs Original Medical Codes
  14. Sequential Encounter Information vs Aggregated Statistics
  15. Performance in the Exposed Group
  16. ...and 2 more sections

Figures (1)

  • Figure 1: The prompt consists of the task instruction for LLM followed by the patient's medical history. The medical history includes demographic information, visit/encounter details, and prescriptions. Both encounters and prescriptions are presented in chronological order, with the earliest first and the latest last. Each encounter contains diagnoses and procedures performed during the visit. The objective of the LLM is to predict the likelihood of a drug overdose within the next 7 and 30 days based on past encounters.