Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Reasoning LLMs Enhance Clinical Document Classification?

Akram Mustafa, Usman Naseem, Mostafa Rahimi Azghadi

TL;DR

The paper investigates whether reasoning LLMs can improve ICD-10 coding for clinical discharge summaries by comparing four reasoning and four non-reasoning models on a balanced MIMIC-IV-derived dataset preprocessed with cTAKES. It finds that reasoning models generally achieve higher accuracy and F1, with Gemini 2.0 Flash Thinking reaching 75% accuracy and 76% F1, while non-reasoning models show greater consistency across repeated runs. The results reveal a trade-off between accuracy and reliability, with well-defined codes showing strong performance and abstract codes exhibiting instability, suggesting a hybrid ensemble approach. The work highlights the need for domain-specific fine-tuning, multi-label extensions, and ensemble strategies to translate AI-assisted coding into robust clinical practice.

Abstract

Clinical document classification is essential for converting unstructured medical texts into standardised ICD-10 diagnoses, yet it faces challenges due to complex medical language, privacy constraints, and limited annotated datasets. Large Language Models (LLMs) offer promising improvements in accuracy and efficiency for this task. This study evaluates the performance and consistency of eight LLMs; four reasoning (Qwen QWQ, Deepseek Reasoner, GPT o3 Mini, Gemini 2.0 Flash Thinking) and four non-reasoning (Llama 3.3, GPT 4o Mini, Gemini 2.0 Flash, Deepseek Chat); in classifying clinical discharge summaries using the MIMIC-IV dataset. Using cTAKES to structure clinical narratives, models were assessed across three experimental runs, with majority voting determining final predictions. Results showed that reasoning models outperformed non-reasoning models in accuracy (71% vs 68%) and F1 score (67% vs 60%), with Gemini 2.0 Flash Thinking achieving the highest accuracy (75%) and F1 score (76%). However, non-reasoning models demonstrated greater stability (91% vs 84% consistency). Performance varied across ICD-10 codes, with reasoning models excelling in complex cases but struggling with abstract categories. Findings indicate a trade-off between accuracy and consistency, suggesting that a hybrid approach could optimise clinical coding. Future research should explore multi-label classification, domain-specific fine-tuning, and ensemble methods to enhance model reliability in real-world applications.

Can Reasoning LLMs Enhance Clinical Document Classification?

TL;DR

The paper investigates whether reasoning LLMs can improve ICD-10 coding for clinical discharge summaries by comparing four reasoning and four non-reasoning models on a balanced MIMIC-IV-derived dataset preprocessed with cTAKES. It finds that reasoning models generally achieve higher accuracy and F1, with Gemini 2.0 Flash Thinking reaching 75% accuracy and 76% F1, while non-reasoning models show greater consistency across repeated runs. The results reveal a trade-off between accuracy and reliability, with well-defined codes showing strong performance and abstract codes exhibiting instability, suggesting a hybrid ensemble approach. The work highlights the need for domain-specific fine-tuning, multi-label extensions, and ensemble strategies to translate AI-assisted coding into robust clinical practice.

Abstract

Clinical document classification is essential for converting unstructured medical texts into standardised ICD-10 diagnoses, yet it faces challenges due to complex medical language, privacy constraints, and limited annotated datasets. Large Language Models (LLMs) offer promising improvements in accuracy and efficiency for this task. This study evaluates the performance and consistency of eight LLMs; four reasoning (Qwen QWQ, Deepseek Reasoner, GPT o3 Mini, Gemini 2.0 Flash Thinking) and four non-reasoning (Llama 3.3, GPT 4o Mini, Gemini 2.0 Flash, Deepseek Chat); in classifying clinical discharge summaries using the MIMIC-IV dataset. Using cTAKES to structure clinical narratives, models were assessed across three experimental runs, with majority voting determining final predictions. Results showed that reasoning models outperformed non-reasoning models in accuracy (71% vs 68%) and F1 score (67% vs 60%), with Gemini 2.0 Flash Thinking achieving the highest accuracy (75%) and F1 score (76%). However, non-reasoning models demonstrated greater stability (91% vs 84% consistency). Performance varied across ICD-10 codes, with reasoning models excelling in complex cases but struggling with abstract categories. Findings indicate a trade-off between accuracy and consistency, suggesting that a hybrid approach could optimise clinical coding. Future research should explore multi-label classification, domain-specific fine-tuning, and ensemble methods to enhance model reliability in real-world applications.

Paper Structure

This paper contains 22 sections, 12 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Clinical Text Classification
  4. Tokenization & Preprocessing in Clinical NLP
  5. Large Language Models for Clinical Coding
  6. Materials and Methods
  7. Dataset Preparation
  8. Preprocessing
  9. Experimental Setup
  10. Evaluation Metrics
  11. Results
  12. LLMs Performance
  13. Consistency Analysis
  14. Detailed Performance Analysis of Various LLMs
  15. Discussion
  16. ...and 7 more sections

Figures (12)

  • Figure 1: Workflow of cTAKES based Tokenization and Repeated LLM Classification for ICD-10 coded Discharge Summaries.
  • Figure 2: Average performance of 4 reasoning vs 4 non-reasoning LLMs in classifying clinical documents.
  • Figure 3: Consistency and F1 Score by Model for A41.
  • Figure 4: Consistency and F1 Score by Model for E78.
  • Figure 5: Consistency and F1 Score by Model for I10.
  • ...and 7 more figures