MELON: Multimodal Mixture-of-Experts with Spectral-Temporal Fusion for Long-Term Mobility Estimation in Critical Care

TL;DR

This work tackles long-term mobility estimation in the ICU using accelerometer data, a setting where raw time-series modeling is challenging and current long-range assessment methods are limited. It introduces MELON, a dual-branch multimodal framework that fuse spectrogram-based spectral features with temporal statistical sequences via a ResNet image encoder and a Time-MoE sequence encoder, culminating in a self-attention fusion and classifier for predicting 12-hour mobility status. Across a multicenter ICU dataset of 126 patients, MELON outperforms baselines and ablations, achieving an overall AUROC of 0.82 and showing strong performance when using wrist-worn data alone, with robust per-class results and a notable link between mobility and brain status. The approach promises practical impact by enabling continuous, low-burden mobility monitoring in critical care, with potential extensions to delirium detection and broader ICU deployment through larger pretraining and additional modalities.

Abstract

Patient mobility monitoring in intensive care is critical for ensuring timely interventions and improving clinical outcomes. While accelerometry-based sensor data are widely adopted in training artificial intelligence models to estimate patient mobility, existing approaches face two key limitations highlighted in clinical practice: (1) modeling the long-term accelerometer data is challenging due to the high dimensionality, variability, and noise, and (2) the absence of efficient and robust methods for long-term mobility assessment. To overcome these challenges, we introduce MELON, a novel multimodal framework designed to predict 12-hour mobility status in the critical care setting. MELON leverages the power of a dual-branch network architecture, combining the strengths of spectrogram-based visual representations and sequential accelerometer statistical features. MELON effectively captures global and fine-grained mobility patterns by integrating a pre-trained image encoder for rich frequency-domain feature extraction and a Mixture-of-Experts encoder for sequence modeling. We trained and evaluated the MELON model on the multimodal dataset of 126 patients recruited from nine Intensive Care Units at the University of Florida Health Shands Hospital main campus in Gainesville, Florida. Experiments showed that MELON outperforms conventional approaches for 12-hour mobility status estimation with an overall area under the receiver operating characteristic curve (AUROC) of 0.82 (95\%, confidence interval 0.78-0.86). Notably, our experiments also revealed that accelerometer data collected from the wrist provides robust predictive performance compared with data from the ankle, suggesting a single-sensor solution that can reduce patient burden and lower deployment costs...

Figures (2)

• Figure 1: In the data preparation stage (a), we collected data from two accelerometers positioned at the patient's nondominant wrist and ankle. Data was then deidentified and stored in a secure server. We generated spectrograms using the Short-Time Fourier Transform and also extracted five features per minute with a 30-second overlap over 12 hours. The MELON model has a dual-branch structure, as shown in (b). The embeddings from the two branches are then fed into a self-attention layer to generate fusion embeddings. Then, the fused embeddings are processed in the classifier for mobility status prediction.
• Figure 2: Violin plot depicting the distribution of Braden Mobility Scores across three different brain statuses: "coma" "delirium" and "normal".