Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal AI on Wound Images and Clinical Notes for Home Patient Referral

Reza Saadati Fard, Emmanuel Agu, Palawat Busaranuvong, Deepak Kumar, Shefalika Gautam, Bengisu Tulu, Diane Strong

TL;DR

This work tackles the challenge of timely and accurate referral decisions for chronic wounds in home care. It introduces DM-WAT, a multimodal framework that combines wound images processed by a Vision Transformer (DeiT-Base-Distilled) with clinical notes encoded by DeBERTa-base, fused via an intermediate strategy and classified with an SVM. By augmenting both modalities and leveraging transfer learning, DM-WAT achieves 77% accuracy and 70% F1, outperforming single-modality and prior multimodal approaches, with Score-CAM and Captum providing interpretable relevance maps for images and notes. The approach holds promise for supporting non-specialist visiting nurses, improving referral timeliness and patient outcomes, while also outlining pathways to further improve data efficiency, augmentation realism, and explainability in real-world deployments.

Abstract

Chronic wounds affect 8.5 million Americans, particularly the elderly and patients with diabetes. These wounds can take up to nine months to heal, making regular care essential to ensure healing and prevent severe outcomes like limb amputations. Many patients receive care at home from visiting nurses with varying levels of wound expertise, leading to inconsistent care. Problematic, non-healing wounds should be referred to wound specialists, but referral decisions in non-clinical settings are often erroneous, delayed, or unnecessary. This paper introduces the Deep Multimodal Wound Assessment Tool (DM-WAT), a machine learning framework designed to assist visiting nurses in deciding whether to refer chronic wound patients. DM-WAT analyzes smartphone-captured wound images and clinical notes from Electronic Health Records (EHRs). It uses DeiT-Base-Distilled, a Vision Transformer (ViT), to extract visual features from images and DeBERTa-base to extract text features from clinical notes. DM-WAT combines visual and text features using an intermediate fusion approach. To address challenges posed by a small and imbalanced dataset, it integrates image and text augmentation with transfer learning to achieve high performance. In evaluations, DM-WAT achieved 77% with std 3% accuracy and a 70% with std 2% F1 score, outperforming prior approaches. Score-CAM and Captum interpretation algorithms provide insights into specific parts of image and text inputs that influence recommendations, enhancing interpretability and trust.

Multimodal AI on Wound Images and Clinical Notes for Home Patient Referral

TL;DR

This work tackles the challenge of timely and accurate referral decisions for chronic wounds in home care. It introduces DM-WAT, a multimodal framework that combines wound images processed by a Vision Transformer (DeiT-Base-Distilled) with clinical notes encoded by DeBERTa-base, fused via an intermediate strategy and classified with an SVM. By augmenting both modalities and leveraging transfer learning, DM-WAT achieves 77% accuracy and 70% F1, outperforming single-modality and prior multimodal approaches, with Score-CAM and Captum providing interpretable relevance maps for images and notes. The approach holds promise for supporting non-specialist visiting nurses, improving referral timeliness and patient outcomes, while also outlining pathways to further improve data efficiency, augmentation realism, and explainability in real-world deployments.

Abstract

Chronic wounds affect 8.5 million Americans, particularly the elderly and patients with diabetes. These wounds can take up to nine months to heal, making regular care essential to ensure healing and prevent severe outcomes like limb amputations. Many patients receive care at home from visiting nurses with varying levels of wound expertise, leading to inconsistent care. Problematic, non-healing wounds should be referred to wound specialists, but referral decisions in non-clinical settings are often erroneous, delayed, or unnecessary. This paper introduces the Deep Multimodal Wound Assessment Tool (DM-WAT), a machine learning framework designed to assist visiting nurses in deciding whether to refer chronic wound patients. DM-WAT analyzes smartphone-captured wound images and clinical notes from Electronic Health Records (EHRs). It uses DeiT-Base-Distilled, a Vision Transformer (ViT), to extract visual features from images and DeBERTa-base to extract text features from clinical notes. DM-WAT combines visual and text features using an intermediate fusion approach. To address challenges posed by a small and imbalanced dataset, it integrates image and text augmentation with transfer learning to achieve high performance. In evaluations, DM-WAT achieved 77% with std 3% accuracy and a 70% with std 2% F1 score, outperforming prior approaches. Score-CAM and Captum interpretation algorithms provide insights into specific parts of image and text inputs that influence recommendations, enhancing interpretability and trust.
Paper Structure (30 sections, 10 equations, 10 figures, 4 tables)

This paper contains 30 sections, 10 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Image-based machine learning for wound assessment
  4. Text-based machine learning for wound assessment
  5. Multimodal machine learning for wound referral recommendation
  6. Methodology
  7. Dataset
  8. Challenges and strategies to mitigate dataset issues
  9. DM-WAT architecture
  10. Image augmentation
  11. Text augmentation
  12. Feature extraction
  13. Visual feature extraction utilizing DeiT-Base-Distilled
  14. Textual feature extraction utilizing DeBERTa-base
  15. Multimodal fusion
  16. ...and 15 more sections

Figures (10)

  • Figure 1: Example of Wound Image from Dataset. Each image corresponds to one of the three referral decision categories along with descriptive clinical notes 4nguyen2020machine.
  • Figure 2: Expert Decision Analysis(A) The bar chart illustrates the imbalance in referral decisions, with most cases falling under urgent referral. (B) The confusion matrix displays agreement (diagonal) and disagreement (off-diagonal) between the experts' decisions, highlighting inconsistencies in the labeling.
  • Figure 3: DM-WAT framework: (A) Data augmentation, (B) feature extraction using deep neural networks, (C) intermediate fusion of features, and (D) classification into three referral categories: continue treatment, non-urgent referral, or urgent referral.
  • Figure 4: Image augmentation operations with visual examples
  • Figure 5: Example of GPT-4 generated text for Wound Descriptions
  • ...and 5 more figures