Explainable, Multi-modal Wound Infection Classification from Images Augmented with Generated Captions

TL;DR

This work addresses infection detection in Diabetic Foot Ulcers (DFUs) by introducing SCARWID, a multimodal retrieval framework that augments wound images with GPT-4o-generated captions to form cross-modal embeddings for infection classification. A dedicated Wound-BLIP captioning model is trained on caption data, while a cross-modal image-text fusion module enables retrieval of top-$k$ similar labeled examples to determine infection status, with $k=5$ by default. Synthetic data from a latent diffusion model further enhances training, boosting robustness and performance across strong baselines; SCARWID with Image+Text achieves ACC around $0.814$ and SEN around $0.852$, outperforming prior methods. Explainability is built in via Grad-CAM, attention rollouts, and cross-modal embedding visualizations, aiding nurses in interpreting predictions and aligning them with medical knowledge. The results suggest practical benefits for point-of-care wound assessment and potential extension to other medical-imaging classification tasks, though limitations such as caption hallucinations and dataset quality warrant careful validation.

Abstract

Infections in Diabetic Foot Ulcers (DFUs) can cause severe complications, including tissue death and limb amputation, highlighting the need for accurate, timely diagnosis. Previous machine learning methods have focused on identifying infections by analyzing wound images alone, without utilizing additional metadata such as medical notes. In this study, we aim to improve infection detection by introducing Synthetic Caption Augmented Retrieval for Wound Infection Detection (SCARWID), a novel deep learning framework that leverages synthetic textual descriptions to augment DFU images. SCARWID consists of two components: (1) Wound-BLIP, a Vision-Language Model (VLM) fine-tuned on GPT-4o-generated descriptions to synthesize consistent captions from images; and (2) an Image-Text Fusion module that uses cross-attention to extract cross-modal embeddings from an image and its corresponding Wound-BLIP caption. Infection status is determined by retrieving the top-k similar items from a labeled support set. To enhance the diversity of training data, we utilized a latent diffusion model to generate additional wound images. As a result, SCARWID outperformed state-of-the-art models, achieving average sensitivity, specificity, and accuracy of 0.85, 0.78, and 0.81, respectively, for wound infection classification. Displaying the generated captions alongside the wound images and infection detection results enhances interpretability and trust, enabling nurses to align SCARWID outputs with their medical knowledge. This is particularly valuable when wound notes are unavailable or when assisting novice nurses who may find it difficult to identify visual attributes of wound infection.

Figures (11)

• Figure 1: A comprehensive solution for identifying infection in wound images along with annotations
• Figure 2: Overview of the Wound-BLIP Architecture. The model uses a wound Image Encoder to process a wound image and then uses an Image-grounded Text Decoder to generate a concise description of the wound. To enhance interpretability, an Image-grounded Text Encoder is utilized to visualize text localization via Grad-CAM heatmaps based on synthetic wound descriptions.
• Figure 3: SCARWID Pipeline at Test Time: The infection classification starts by considering a query wound image $I^q_i$ as an input. After that Wound-BLIP generates a wound description $T^q_i$ corresponding to $I^q_i$. Then the Image-Text Fusion model, $F_\theta$, takes both $I^q$ and $T^q$ as inputs and transforms them into a $d$-dimensional multimodal embedding vector. Then the framework retrieves the top $k$-nearest neighbor objects in embedding spaces from the support document $\mathcal{D}_{support}$. Finally, the predicted status of the input $I^q_i$ is determined by the most common labels of $k$ retrieved objects $\text{mode}(Y^{s}_{j_{\text{top-k}}})$. Where, $j_{\text{top-k}}$ denotes top-$k$ indices.
• Figure 4: Label-Guided Prompting for Textual Metadata Generation: {System: You are a wound care physician}. A user's prompt is as follows. {User: $<$image$>$ | The wound image was labeled as $<$label$>$. Please briefly describe the image in 1 sentence.}
• Figure 5: Examples of Conditional Synthesized Wound Images by the Diffusion Model