Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applications

Jean-Philippe Corbeil, Asma Ben Abacha, George Michalopoulos, Phillip Swazinna, Miguel Del-Agua, Jerome Tremblay, Akila Jeeson Daniel, Cari Bader, Yu-Cheng Cho, Pooja Krishnan, Nathan Bodenstab, Thomas Lin, Wenxuan Teng, Francois Beaulieu, Paul Vozila

TL;DR

This work tackles two high-impact clinical NLP tasks—structured nursing observation extraction from nurse dictations and medical order extraction from doctor–patient conversations—due to data scarcity and sensitivity. It evaluates both open- and closed-weight LLMs on proprietary hospital data and newly released open datasets, and introduces an agentic data-generation pipeline to create realistic, non-sensitive nursing data (SYNUR) alongside the first medical order extraction dataset (SIMORD). The authors demonstrate that LLMs can reduce documentation burden, analyze the relative strengths of different model families, and highlight practical challenges such as long-context handling and output parsing. The release of SYNUR and SIMORD provides valuable resources for ongoing research, while the study outlines a path toward scalable, LLM-driven clinical documentation solutions, with clear avenues for future work including larger datasets and constrained-output improvements.

Abstract

Large language models (LLMs) such as GPT-4o and o1 have demonstrated strong performance on clinical natural language processing (NLP) tasks across multiple medical benchmarks. Nonetheless, two high-impact NLP tasks - structured tabular reporting from nurse dictations and medical order extraction from doctor-patient consultations - remain underexplored due to data scarcity and sensitivity, despite active industry efforts. Practical solutions to these real-world clinical tasks can significantly reduce the documentation burden on healthcare providers, allowing greater focus on patient care. In this paper, we investigate these two challenging tasks using private and open-source clinical datasets, evaluating the performance of both open- and closed-weight LLMs, and analyzing their respective strengths and limitations. Furthermore, we propose an agentic pipeline for generating realistic, non-sensitive nurse dictations, enabling structured extraction of clinical observations. To support further research in both areas, we release SYNUR and SIMORD, the first open-source datasets for nurse observation extraction and medical order extraction.

Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applications

TL;DR

This work tackles two high-impact clinical NLP tasks—structured nursing observation extraction from nurse dictations and medical order extraction from doctor–patient conversations—due to data scarcity and sensitivity. It evaluates both open- and closed-weight LLMs on proprietary hospital data and newly released open datasets, and introduces an agentic data-generation pipeline to create realistic, non-sensitive nursing data (SYNUR) alongside the first medical order extraction dataset (SIMORD). The authors demonstrate that LLMs can reduce documentation burden, analyze the relative strengths of different model families, and highlight practical challenges such as long-context handling and output parsing. The release of SYNUR and SIMORD provides valuable resources for ongoing research, while the study outlines a path toward scalable, LLM-driven clinical documentation solutions, with clear avenues for future work including larger datasets and constrained-output improvements.

Abstract

Large language models (LLMs) such as GPT-4o and o1 have demonstrated strong performance on clinical natural language processing (NLP) tasks across multiple medical benchmarks. Nonetheless, two high-impact NLP tasks - structured tabular reporting from nurse dictations and medical order extraction from doctor-patient consultations - remain underexplored due to data scarcity and sensitivity, despite active industry efforts. Practical solutions to these real-world clinical tasks can significantly reduce the documentation burden on healthcare providers, allowing greater focus on patient care. In this paper, we investigate these two challenging tasks using private and open-source clinical datasets, evaluating the performance of both open- and closed-weight LLMs, and analyzing their respective strengths and limitations. Furthermore, we propose an agentic pipeline for generating realistic, non-sensitive nurse dictations, enabling structured extraction of clinical observations. To support further research in both areas, we release SYNUR and SIMORD, the first open-source datasets for nurse observation extraction and medical order extraction.

Paper Structure

This paper contains 32 sections, 6 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Nursing
  4. Medical Order Extraction
  5. Data Creation Methods
  6. Nursing Datasets
  7. Proprietary Nursing Datasets
  8. SYNUR: Synthetic Nursing Dataset
  9. Quantitative Assessments of SYNUR
  10. Statistics
  11. LLM-as-a-Judge
  12. Inter-Annotator Agreement
  13. Medical Order Dataset
  14. Doctor-Patient Consultation Datasets
  15. SIMORD: Simulated Orders Dataset
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Examples of our two real-world unstructured-to-structured tasks in clinical settings on spoken transcriptions from healthcare practitioners: A) medical order extraction from doctor-patient dialog, and B) observation extraction from nurse dictation. Icon attributions in Appendix \ref{['sec:icon']}.
  • Figure 2: SYNUR Dataset Creation Pipeline for generating realistic, synthetic nurse dictations with expert nurse annotations, comprising six steps: starting from 80 human-verified, fabricated dictations to final expert annotations. The number below each step represents the final amount of output elements generated at this specific step. The output of one step is the input of the next step, except for step 5 for which we also include 5 randomly sampled examples from the seed dictations. At step 6, the annotators also have access to the schema of step 3 along the synthetic dictation to produce its gold annotations.
  • Figure 3: Three-step method for the nurse observation extraction task.