CNSight: Evaluation of Clinical Note Segmentation Tools

TL;DR

This paper tackles automatic segmentation of clinical notes into distinct sections to enable downstream analytics such as information extraction and cohort identification. It benchmarks a spectrum of approaches—from rule-based baselines and domain-specific transformers to API-based large language models—on a curated MIMIC-IV dataset under the CNSight pipeline. The results show that GPT-5-mini achieves the best overall performance, with traditional baselines performing well on structured sentence tasks and human annotators displaying flexible judgments on ambiguous content. These findings guide method selection for clinical note structuring and lay groundwork for reliable downstream clinical informatics tasks.

Abstract

Clinical notes are often stored in unstructured or semi-structured formats after extraction from electronic medical record (EMR) systems, which complicates their use for secondary analysis and downstream clinical applications. Reliable identification of section boundaries is a key step toward structuring these notes, as sections such as history of present illness, medications, and discharge instructions each provide distinct clinical contexts. In this work, we evaluate rule-based baselines, domain-specific transformer models, and large language models for clinical note segmentation using a curated dataset of 1,000 notes from MIMIC-IV. Our experiments show that large API-based models achieve the best overall performance, with GPT-5-mini reaching a best average F1 of 72.4 across sentence-level and freetext segmentation. Lightweight baselines remain competitive on structured sentence-level tasks but falter on unstructured freetext. Our results provide guidance for method selection and lay the groundwork for downstream tasks such as information extraction, cohort identification, and automated summarization.

Figures (5)

• Figure 1: LLMs enable automated segmentation of clinical notes. Overview of the CNSight pipeline, where free-text or sentence-split clinical notes are processed by large language models and baseline methods to extract structured clinical sections such as patient history, chief complaint, medications, and discharge instructions.
• Figure 2: Summary of metrics across models. Comparison of precision, recall, and F1 scores across medical and general-purpose language models. General-purpose models outperform smaller domain-specific models.
• Figure 3: A small set of clinical tags dominates the dataset. Bar chart showing the most frequent clinical tags (occurring at least 50 times), with counts on the y-axis and tag categories on the x-axis.
• Figure 4: Most notes contained approximately 18 tags. Bar chart showing the distribution of the number of tags across all notes, shows a relatively consistent number across the dataset.
• Figure 5: Sentence length variation. Bar chart showing the sentence length distributions across tags, shows a high number of short "sentences", referring to phrases and small extracted header sections for increased precision.