Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data

Aram Ansary Ogholbake, Hannah Choi, Spencer Brandenburg, Alyssa Antuna, Zahraa Al-Sharshahi, Makayla Cox, Haseeb Ahmed, Jacqueline Frank, Nathan Millson, Luke Bauerle, Jessica Lee, David Dornbos, Qiang Cheng

Abstract

We propose AttentionMixer, a unified deep learning framework for multimodal detection of brain edema that combines structural head CT (HCT) with routine clinical metadata. While HCT provides rich spatial information, clinical variables such as age, laboratory values, and scan timing capture complementary context that might be ignored or naively concatenated. AttentionMixer is designed to fuse these heterogeneous sources in a principled and efficient manner. HCT volumes are first encoded using a self-supervised Vision Transformer Autoencoder (ViT-AE++), without requiring large labeled datasets. Clinical metadata are mapped into the same feature space and used as keys and values in a cross-attention module, where HCT-derived feature vector serves as queries. This cross-attention fusion allows the network to dynamically modulate imaging features based on patient-specific context and provides an interpretable mechanism for multimodal integration. A lightweight MLP-Mixer then refines the fused representation before final classification, enabling global dependency modeling with substantially reduced parameter overhead. Missing or incomplete metadata are handled via a learnable embedding, promoting robustness to real-world clinical data quality. We evaluate AttentionMixer on a curated brain HCT cohort with expert edema annotations using five-fold cross-validation. Compared with strong HCT-only, metadata-only, and prior multimodal baselines, AttentionMixer achieves superior performance (accuracy 87.32%, precision 92.10%, F1-score 85.37%, AUC 94.14%). Ablation studies confirm the benefit of both cross-attention and MLP-Mixer refinement, and permutation-based metadata importance analysis highlights clinically meaningful variables driving predictions. These results demonstrate that structured, interpretable multimodal fusion can substantially improve edema detection in clinical practice.

A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data

Abstract

We propose AttentionMixer, a unified deep learning framework for multimodal detection of brain edema that combines structural head CT (HCT) with routine clinical metadata. While HCT provides rich spatial information, clinical variables such as age, laboratory values, and scan timing capture complementary context that might be ignored or naively concatenated. AttentionMixer is designed to fuse these heterogeneous sources in a principled and efficient manner. HCT volumes are first encoded using a self-supervised Vision Transformer Autoencoder (ViT-AE++), without requiring large labeled datasets. Clinical metadata are mapped into the same feature space and used as keys and values in a cross-attention module, where HCT-derived feature vector serves as queries. This cross-attention fusion allows the network to dynamically modulate imaging features based on patient-specific context and provides an interpretable mechanism for multimodal integration. A lightweight MLP-Mixer then refines the fused representation before final classification, enabling global dependency modeling with substantially reduced parameter overhead. Missing or incomplete metadata are handled via a learnable embedding, promoting robustness to real-world clinical data quality. We evaluate AttentionMixer on a curated brain HCT cohort with expert edema annotations using five-fold cross-validation. Compared with strong HCT-only, metadata-only, and prior multimodal baselines, AttentionMixer achieves superior performance (accuracy 87.32%, precision 92.10%, F1-score 85.37%, AUC 94.14%). Ablation studies confirm the benefit of both cross-attention and MLP-Mixer refinement, and permutation-based metadata importance analysis highlights clinically meaningful variables driving predictions. These results demonstrate that structured, interpretable multimodal fusion can substantially improve edema detection in clinical practice.

Paper Structure

This paper contains 21 sections, 15 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Formulation
  5. Preprocessing of HCT and Metadata
  6. HCT Feature Extraction with Vision Transformer Autoencoder
  7. Metadata Processing and Embedding
  8. Cross-Modal Attention Fusion
  9. Multi-head cross-attention.
  10. Fusion for downstream processing.
  11. Feature Mixing and Classification
  12. Training Objective
  13. Results
  14. Experimental Setup and Implementation Details
  15. Overall Performance
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Overall diagram of AttentionMixer. HCT volumes are encoded by a self-supervised ViT-based encoder, while tabular metadata are embedded into the same latent space. A cross-attention module uses HCT features as queries and metadata as keys/values to produce a fused representation, which is further refined by an MLP-Mixer before classification.
  • Figure 2: Average ROC curve of AttentionMixer across 5-fold cross-validation. The mean AUC is 94%.
  • Figure 3: Top-10 metadata variables ranked by permutation feature importance. Importance is measured as the mean decrease in AUC ($\Delta \text{AUC}$) across five cross-validation folds when each feature is independently permuted. Higher values indicate stronger contribution of the metadata feature to the multimodal edema classification model.
  • Figure 4: Distribution of predicted probabilities for edema and non-edema cases.