Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Medical Triage as Pairwise Ranking: A Benchmark for Urgency in Patient Portal Messages

Joseph Gatto, Parker Seegmiller, Timothy Burdick, Philip Resnik, Roshnik Rahat, Sarah DeLozier, Sarah M. Preum

TL;DR

This work reframes the triage of asynchronous patient portal messages as a pairwise urgency ranking problem and introduces PMR-Bench, a large-scale, multi-source dataset with in-domain EHR context. It proposes two fines-tuning strategies, UrgentSFT and UrgentReward, and demonstrates that pairwise approaches yield superior inbox-sorting performance across multiple datasets and model scales, including lightweight 8B variants. The intrinsic and extrinsic evaluations show clear gains over baselines and highlight data-efficiency and stability advantages of the proposed methods. The study provides public data and actionable insights toward safer, faster prioritization of patient messages in clinical workflows, with attention to privacy and ethical considerations.

Abstract

Medical triage is the task of allocating medical resources and prioritizing patients based on medical need. This paper introduces the first large-scale public dataset for studying medical triage in the context of asynchronous outpatient portal messages. Our novel task formulation views patient message triage as a pairwise inference problem, where we train LLMs to choose `"which message is more medically urgent" in a head-to-head tournament-style re-sort of a physician's inbox. Our novel benchmark PMR-Bench contains 1569 unique messages and 2,000+ high-quality test pairs for pairwise medical urgency assessment alongside a scalable training data generation pipeline. PMR-Bench includes samples that contain both unstructured patient-written messages alongside real electronic health record (EHR) data, emulating a real-world medical triage scenario. We develop a novel automated data annotation strategy to provide LLMs with in-domain guidance on this task. The resulting data is used to train two model classes, UrgentReward and UrgentSFT, leveraging Bradley-Terry and next token prediction objective, respectively to perform pairwise urgency classification. We find that UrgentSFT achieves top performance on PMR-Bench, with UrgentReward showing distinct advantages in low-resource settings. For example, UrgentSFT-8B and UrgentReward-8B provide a 15- and 16-point boost, respectively, on inbox sorting metrics over off-the-shelf 8B models. Paper resources can be found at https://tinyurl.com/Patient-Message-Triage

Medical Triage as Pairwise Ranking: A Benchmark for Urgency in Patient Portal Messages

TL;DR

This work reframes the triage of asynchronous patient portal messages as a pairwise urgency ranking problem and introduces PMR-Bench, a large-scale, multi-source dataset with in-domain EHR context. It proposes two fines-tuning strategies, UrgentSFT and UrgentReward, and demonstrates that pairwise approaches yield superior inbox-sorting performance across multiple datasets and model scales, including lightweight 8B variants. The intrinsic and extrinsic evaluations show clear gains over baselines and highlight data-efficiency and stability advantages of the proposed methods. The study provides public data and actionable insights toward safer, faster prioritization of patient messages in clinical workflows, with attention to privacy and ethical considerations.

Abstract

Medical triage is the task of allocating medical resources and prioritizing patients based on medical need. This paper introduces the first large-scale public dataset for studying medical triage in the context of asynchronous outpatient portal messages. Our novel task formulation views patient message triage as a pairwise inference problem, where we train LLMs to choose `"which message is more medically urgent" in a head-to-head tournament-style re-sort of a physician's inbox. Our novel benchmark PMR-Bench contains 1569 unique messages and 2,000+ high-quality test pairs for pairwise medical urgency assessment alongside a scalable training data generation pipeline. PMR-Bench includes samples that contain both unstructured patient-written messages alongside real electronic health record (EHR) data, emulating a real-world medical triage scenario. We develop a novel automated data annotation strategy to provide LLMs with in-domain guidance on this task. The resulting data is used to train two model classes, UrgentReward and UrgentSFT, leveraging Bradley-Terry and next token prediction objective, respectively to perform pairwise urgency classification. We find that UrgentSFT achieves top performance on PMR-Bench, with UrgentReward showing distinct advantages in low-resource settings. For example, UrgentSFT-8B and UrgentReward-8B provide a 15- and 16-point boost, respectively, on inbox sorting metrics over off-the-shelf 8B models. Paper resources can be found at https://tinyurl.com/Patient-Message-Triage
Paper Structure (42 sections, 1 equation, 5 figures, 13 tables)

This paper contains 42 sections, 1 equation, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for Document Ranking:
  4. Patient Messages Triaging
  5. Methods
  6. PMR as a Pairwise Ranking Task
  7. PMR-Bench Dataset Overview
  8. Urgency Annotation
  9. Dataset Details
  10. UrgentSFT
  11. UrgentReward
  12. Experimental Setup
  13. Intrinsic Evaluation
  14. Extrinsic Evaluation
  15. Data Preparation:
  16. ...and 27 more sections

Figures (5)

  • Figure 1: In this study, we introduce PMR-Bench, a novel dataset for evaluating LLM capacity to produce "Urgency Aware" inboxes, where patient messages in clinicians' inboxes are sorted by medical urgency. Note that in a categorical setup, multiple messages can have a similar level of urgency.
  • Figure 2: Example data format for PMR-Reddit and PMR-Synth/Real. Data from Reddit is unstructured text, with responses from moderator-verified clinical experts. Data from PMR-Synth and PMR-Real include linked EHR data, as the EHR data impact triage decisions.
  • Figure 3: Shown is the system prompt provided to all models in all experiments
  • Figure 4: The prompt used for UrgentSFT, Instruct, and Reasoning Baselines.
  • Figure 5: The prompt used for UrgentReward. This prompt differs as we leverage pre-trained reward models which are trained to score completions. We thus re-formulate the task to better utilize existing knowledge.