TocBERT: Medical Document Structure Extraction Using Bidirectional Transformers

TL;DR

The paper tackles hierarchical text segmentation in unformatted medical documents by detecting titles and subtitles to improve data cleaning and targeted information retrieval. It introduces TocBERT, a NER-based method built on fine-tuned Bio-ClinicalBERT, trained on semi-automatically labeled MIMIC-III discharge summaries and evaluated against a rule-based TocRegex baseline. TocBERT achieves a hierarchical F1 of 0.728 (versus 0.606 for TocRegex) and a linear F1 of 0.846 (slightly higher than TocRegex), demonstrating strong use of semantic context to distinguish titles from subtitles. The work enables improved domain adaptation and the potential construction of a topics ontology from detected titles, with practical implications for medical document processing and retrieval.

Abstract

Text segmentation holds paramount importance in the field of Natural Language Processing (NLP). It plays an important role in several NLP downstream tasks like information retrieval and document summarization. In this work, we propose a new solution, namely TocBERT, for segmenting texts using bidirectional transformers. TocBERT represents a supervised solution trained on the detection of titles and sub-titles from their semantic representations. This task was formulated as a named entity recognition (NER) problem. The solution has been applied on a medical text segmentation use-case where the Bio-ClinicalBERT model is fine-tuned to segment discharge summaries of the MIMIC-III dataset. The performance of TocBERT has been evaluated on a human-labeled ground truth corpus of 250 notes. It achieved an F1-score of 84.6% when evaluated on a linear text segmentation problem and 72.8% on a hierarchical text segmentation problem. It outperformed a carefully designed rule-based solution, particularly in distinguishing titles from subtitles.

Figures (6)

• Figure 1: An extract from a discharge summary from the MIMIC-III database annotated with titles (yellow) and subtitles (blue)
• Figure 2: Two-level neural networks for text segmentation: the first-level network builds sentence embeddings from token embeddings while the second-level network labels the sequence of sentence representations as: topic-change $(1)$ or no-topic-change $(0)$
• Figure 3: a) distribution of sub-corpora sizes of the MIMIC-III database. 2) distribution of reports' lengths of the discharge summaries corpus.
• Figure 4: Top $35$ "candidate" titles extracted using the first pattern of the TocRegex solution including true positives (blue) and false positives (light brown)
• Figure 5: Labeling token sequences in TocBERT: Example 1) two titles "Past Medical History" and "Family History" are labeled with "I-title" while other tokens are labeled with "O"; Example 2) one title "Physical exam" is labeled with "I-title" and four subtitles "HEENT", "Neck", "Lungs" and "Extremities" are labeled with "I-Stitle", while other (outside) tokens are labeled with "O".