Towards Relaxed Multimodal Inputs for Gait-based Parkinson's Disease Assessment

TL;DR

This work tackles Parkinson's disease assessment using multimodal gait data, addressing practical challenges of synchronized inputs and modality collapse. It introduces TRIP, a relaxed-input framework that casts multimodal learning as a multi-objective optimization problem, enabling asynchronous training and optional modalities at inference. A margin-based rebalancing strategy mitigates intra- and inter-class imbalance across modalities. Extensive experiments on three public gait datasets show state-of-the-art performance under both asynchronous and synchronous conditions, with strong per-modality accuracy and robust modality-collapse mitigation, highlighting TRIP's potential for real-world clinical deployment.

Abstract

Parkinson's disease assessment has garnered growing interest in recent years, particularly with the advent of sensor data and machine learning techniques. Among these, multimodal approaches have demonstrated strong performance by effectively integrating complementary information from various data sources. However, two major limitations hinder their practical application: (1) the need to synchronize all modalities during training, and (2) the dependence on all modalities during inference. To address these issues, we propose the first Parkinson's assessment system that formulates multimodal learning as a multi-objective optimization (MOO) problem. This not only allows for more flexible modality requirements during both training and inference, but also handles modality collapse issue during multimodal information fusion. In addition, to mitigate the imbalance within individual modalities, we introduce a margin-based class rebalancing strategy to enhance category learning. We conduct extensive experiments on three public datasets under both synchronous and asynchronous settings. The results show that our framework-Towards Relaxed InPuts (TRIP)-achieves state-of-the-art performance, outperforming the best baselines by 16.48, 6.89, and 11.55 percentage points in the asynchronous setting, and by 4.86 and 2.30 percentage points in the synchronous setting, highlighting its effectiveness and adaptability.

Figures (5)

• Figure 1: TRIP overview (relaxed-input scenario). Gait lab setup: conventional labs record multiple synchronized modalities (e.g., 3D motion capture and wearables), but trials often have missing streams or dropouts. Our pipeline supports asynchronous inputs and missing modalities through: (1) Relaxed Inputs: trains and infers with time-misaligned clips and enabled missing modalities (blue bands); each stream is encoded independently, then processed by a shared backbone and modality heads—matching real deployment. (2) Shared-Only MOO: modality losses update their own private parameters, while only the shared parameters $\varphi$ receive mixed, conflict-averse gradients using our proposed MOO objective. (3) Imbalance-Aware Heads: our margin-based rebalancing strategy at each head reweights classes with cosine-normalized logits to handle the inter/intra-class imbalance typical of multimodal gait PD data.
• Figure 2: Illustration of the MOO learning paradigm (two-modalities scenarios). Left: gradient conflict between the update direction of the sensor modality ($g_{\text{sensor}}$) and the overall average gradient direction. Right: conflict resolved after applying MOO optimization, aligning the adjusted modality gradients ($g'_{\text{sensor}}$, $g_{\text{skel}}$) with the shared descent direction.
• Figure 3: Class distribution of multimodal PD datasets (FBG and FOG). Both datasets exhibit long-tailed distributions and noticeable intra-class imbalance between sensor and skeleton modalities.
• Figure 4: Demonstration of effective modality collapse mitigation with TRIP. Results are obtained using WearGait dataset under synchronous inputs condition. Values in x-axis include all combinations of different modalities used during inference (W: Walkway, I: Insole, M: IMU).
• Figure 5: Performances analysis under asynchronous input settings on the FOG dataset with two hyperparameters: $\alpha$ and $m$, which respectively represents the MOO coefficient and additive margin coefficient.