Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving VTE Identification through Language Models from Radiology Reports: A Comparative Study of Mamba, Phi-3 Mini, and BERT

Jamie Deng, Yusen Wu, Yelena Yesha, Phuong Nguyen

TL;DR

The paper tackles automated identification of venous thromboembolism (VTE) from radiology reports, addressing limitations of prior complex pipelines that relied on hand-crafted rule sets. It proposes a Mamba-based classifier using a State Space Model foundation with long-context capability (8K tokens) and end-to-end fine-tuning, eliminating the need for rule-based classification and improving efficiency. Comparative experiments show the Mamba model achieving 97% accuracy and F1 on DVT and 98% accuracy and F1 on PE, outperforming or matching BERT baselines while handling longer texts; a lightweight Phi-3 Mini LLM with QLoRA also performs well but is more computationally demanding. The results support the practicality of long-context, rule-free NLP solutions for critical medical text classification, with potential for deployment in real-world clinical workflows.

Abstract

Venous thromboembolism (VTE) is a critical cardiovascular condition, encompassing deep vein thrombosis (DVT) and pulmonary embolism (PE). Accurate and timely identification of VTE is essential for effective medical care. This study builds upon our previous work, which addressed VTE detection using deep learning methods for DVT and a hybrid approach combining deep learning and rule-based classification for PE. Our earlier approaches, while effective, had two major limitations: they were complex and required expert involvement for feature engineering of the rule set. To overcome these challenges, we utilize the Mamba architecture-based classifier. This model achieves remarkable results, with a 97\% accuracy and F1 score on the DVT dataset and a 98\% accuracy and F1 score on the PE dataset. In contrast to the previous hybrid method on PE identification, the Mamba classifier eliminates the need for hand-engineered rules, significantly reducing model complexity while maintaining comparable performance. Additionally, we evaluated a lightweight Large Language Model (LLM), Phi-3 Mini, in detecting VTE. While this model delivers competitive results, outperforming the baseline BERT models, it proves to be computationally intensive due to its larger parameter set. Our evaluation shows that the Mamba-based model demonstrates superior performance and efficiency in VTE identification, offering an effective solution to the limitations of previous approaches.

Improving VTE Identification through Language Models from Radiology Reports: A Comparative Study of Mamba, Phi-3 Mini, and BERT

TL;DR

The paper tackles automated identification of venous thromboembolism (VTE) from radiology reports, addressing limitations of prior complex pipelines that relied on hand-crafted rule sets. It proposes a Mamba-based classifier using a State Space Model foundation with long-context capability (8K tokens) and end-to-end fine-tuning, eliminating the need for rule-based classification and improving efficiency. Comparative experiments show the Mamba model achieving 97% accuracy and F1 on DVT and 98% accuracy and F1 on PE, outperforming or matching BERT baselines while handling longer texts; a lightweight Phi-3 Mini LLM with QLoRA also performs well but is more computationally demanding. The results support the practicality of long-context, rule-free NLP solutions for critical medical text classification, with potential for deployment in real-world clinical workflows.

Abstract

Venous thromboembolism (VTE) is a critical cardiovascular condition, encompassing deep vein thrombosis (DVT) and pulmonary embolism (PE). Accurate and timely identification of VTE is essential for effective medical care. This study builds upon our previous work, which addressed VTE detection using deep learning methods for DVT and a hybrid approach combining deep learning and rule-based classification for PE. Our earlier approaches, while effective, had two major limitations: they were complex and required expert involvement for feature engineering of the rule set. To overcome these challenges, we utilize the Mamba architecture-based classifier. This model achieves remarkable results, with a 97\% accuracy and F1 score on the DVT dataset and a 98\% accuracy and F1 score on the PE dataset. In contrast to the previous hybrid method on PE identification, the Mamba classifier eliminates the need for hand-engineered rules, significantly reducing model complexity while maintaining comparable performance. Additionally, we evaluated a lightweight Large Language Model (LLM), Phi-3 Mini, in detecting VTE. While this model delivers competitive results, outperforming the baseline BERT models, it proves to be computationally intensive due to its larger parameter set. Our evaluation shows that the Mamba-based model demonstrates superior performance and efficiency in VTE identification, offering an effective solution to the limitations of previous approaches.
Paper Structure (15 sections, 1 equation, 3 figures, 2 tables)

This paper contains 15 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Methods
  3. SSM Fundamentals
  4. Classification Model
  5. Experiment Results
  6. VTE Datasets
  7. Experimental Settings
  8. Model Performance
  9. Related Work
  10. Traditional approaches
  11. Deep Learning and Hybrid methods
  12. Mamba language models
  13. Future Works
  14. Conclusion
  15. Acknowledgement

Figures (3)

  • Figure 1: Structure of Mamba-based classifier. Mamba-130M contains 24 layers.
  • Figure 2: ROC curves of different methods on the DVT dataset of Ultrasound reports. (Class 0 - No acute DVT, Class 1 - Upper extremity acute DVT, and Class 2 - Lower extremity acute DVT.)
  • Figure 3: ROC curves of different methods on the PE dataset of CT scan reports. (class 0 - No PE, class 1 - PE.)