Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts

Karthikeyan K, Raghuveer Thirukovalluru, David Carlson

TL;DR

ClinStructor addresses the challenge of unstructured clinical notes by introducing an LLM-driven pipeline that converts admission notes into structured 50-question, 50-answer features for ICU mortality prediction. It comprises three stages: feature identification via clustering of LLM-generated questions, feature extraction producing a fixed 50 QA representation, and LoRA-based fine-tuning of a predictor. Compared with direct fine-tuning on raw notes, ClinStructor achieves competitive AUC with a modest 2–3 percentage-point drop, while offering explicit, inspectable input features and improved cross-site generalizability. This work lays a foundation for interpretable, robust, and generalizable clinical ML systems, with practical implications for bias control and regulatory compliance.

Abstract

Clinical notes contain valuable, context-rich information, but their unstructured format introduces several challenges, including unintended biases (e.g., gender or racial bias), and poor generalization across clinical settings (e.g., models trained on one EHR system may perform poorly on another due to format differences) and poor interpretability. To address these issues, we present ClinStructor, a pipeline that leverages large language models (LLMs) to convert clinical free-text into structured, task-specific question-answer pairs prior to predictive modeling. Our method substantially enhances transparency and controllability and only leads to a modest reduction in predictive performance (a 2-3% drop in AUC), compared to direct fine-tuning, on the ICU mortality prediction task. ClinStructor lays a strong foundation for building reliable, interpretable, and generalizable machine learning models in clinical environments.

ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts

TL;DR

ClinStructor addresses the challenge of unstructured clinical notes by introducing an LLM-driven pipeline that converts admission notes into structured 50-question, 50-answer features for ICU mortality prediction. It comprises three stages: feature identification via clustering of LLM-generated questions, feature extraction producing a fixed 50 QA representation, and LoRA-based fine-tuning of a predictor. Compared with direct fine-tuning on raw notes, ClinStructor achieves competitive AUC with a modest 2–3 percentage-point drop, while offering explicit, inspectable input features and improved cross-site generalizability. This work lays a foundation for interpretable, robust, and generalizable clinical ML systems, with practical implications for bias control and regulatory compliance.

Abstract

Clinical notes contain valuable, context-rich information, but their unstructured format introduces several challenges, including unintended biases (e.g., gender or racial bias), and poor generalization across clinical settings (e.g., models trained on one EHR system may perform poorly on another due to format differences) and poor interpretability. To address these issues, we present ClinStructor, a pipeline that leverages large language models (LLMs) to convert clinical free-text into structured, task-specific question-answer pairs prior to predictive modeling. Our method substantially enhances transparency and controllability and only leads to a modest reduction in predictive performance (a 2-3% drop in AUC), compared to direct fine-tuning, on the ICU mortality prediction task. ClinStructor lays a strong foundation for building reliable, interpretable, and generalizable machine learning models in clinical environments.

Paper Structure

This paper contains 25 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Overview of Interpretability Techniques
  4. A Broad Overview of Applications of LLMs in Healthcare
  5. Specialized LLMs in Healthcare and Medical domain
  6. LLMs for Feature Importance
  7. Text Bottleneck Model
  8. Our Proposed Method: ClinStructor
  9. Stage 1: Feature Identification
  10. Stage 2: Feature Extraction
  11. Stage 3: Fine-tuning
  12. Experiments
  13. Dataset
  14. Direct fine-tuning vs. ClinStructor fine-tuning
  15. Results
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Illustration of the ClinStructor Pipeline: The proposed pipeline comprises three main stages: (1) Feature Identification, (2) Feature Extraction, and (3) fine-tuning.
  • Figure 2: Effective Number of Questions: The plot shows the distribution of the effective number of questions—i.e., the number of questions with non-"N/A" answers, for both Qwen-32B and LLaMA-70B models. Note that approximately 40% of the answers from LLaMA-70B and 50% from Qwen-32B are "N/A".