From Generative Modeling to Clinical Classification: A GPT-Based Architecture for EHR Notes

TL;DR

The paper tackles the challenge of adapting large language models to unstructured clinical notes under data and compute constraints. It proposes a GPT-2 based decoder architecture for EHR note classification that freezes the majority of parameters and only updates the final Transformer block, the final LayerNorm, and a lightweight classification head, initialized from OpenAI open-weight checkpoints, with a 1024 token context. A transparent parameter and time complexity analysis shows the approach freezes over $94\%$ of parameters, while maintaining strong performance across multi-label and binary CheXpert style tasks on MIMIC-IV radiology notes and uncertainty aware labels. The work demonstrates a scalable, reproducible pathway to deploy large language models in clinical NLP and provides a foundation for efficient domain adaptation in data limited settings.

Abstract

The increasing availability of unstructured clinical narratives in electronic health records (EHRs) has created new opportunities for automated disease characterization, cohort identification, and clinical decision support. However, modeling long, domain-specific clinical text remains challenging due to limited labeled data, severe class imbalance, and the high computational cost of adapting large pretrained language models. This study presents a GPT-based architecture for clinical text classification that adapts a pretrained decoder-only Transformer using a selective fine-tuning strategy. Rather than updating all model parameters, the majority of the GPT-2 backbone is frozen, and training is restricted to the final Transformer block, the final layer normalization, and a lightweight classification head. This approach substantially reduces the number of trainable parameters while preserving the representational capacity required to model complex clinical language. The proposed method is evaluated on radiology reports from the MIMIC-IV-Note dataset using uncertainty-aware CheXpert-style labels derived directly from report text. Experiments cover multiple problem formulations, including multi-label classification of radiographic findings, binary per-label classification under different uncertainty assumptions, and aggregate disease outcome prediction. Across varying dataset sizes, the model exhibits stable convergence behavior and strong classification performance, particularly in settings dominated by non-mention and negated findings. Overall, the results indicate that selective fine-tuning of pretrained generative language models provides an efficient and effective pathway for clinical text classification, enabling scalable adaptation to real-world EHR data while significantly reducing computational complexity.

Figures (15)

• Figure 1: Overall architecture of the GPT model.
• Figure 2: Input Pre-Processing and Embedding Construction in Transformer-Based Language Models.
• Figure 3: Computation Flow of Causal Multi-Head Self-Attention with Scaled Dot-Product Attention, Masking, and Dropout.
• Figure 4: Decoder-only Transformer block with pre-normalization, masked multi-head self-attention, residual connections, dropout, and a position-wise feed-forward network.
• Figure 5: Selective fine-tuning workflow for a GPT-based classification model. Pretrained parameters are loaded and all layers are frozen except the final Transformer block, final LayerNorm, and the task-specific classification head. Fine-tuning proceeds over mini-batches using cross-entropy loss and AdamW updates applied only to trainable parameters, yielding a task-specific GPT classifier.