MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series

TL;DR

MAESTRO addresses multimodal time-series with arbitrary missing modalities by integrating symbolic tokenization, adaptive per-modality sparse attention, cross-modal sparse attention for long sequences, and a loss-free Sparse MoE router. It avoids reliance on a single anchor modality or exhaustive pairwise modeling and scales to many modalities while maintaining robustness and efficiency, showing improvements of roughly $4\%$ to $8\%$ under complete observations and around $9\%$ under partial observations across four real-world datasets. The framework’s components—symbolic tokenization with a missingness symbol, modality-aware budgeting, sparse intra- and cross-modal attention, and dynamic MoE routing—collectively enable learning of task-relevant intra- and inter-modal interactions in dynamic sensing environments. These results demonstrate practical impact for applications in healthcare, activity recognition, and continuous monitoring where sensor availability is heterogeneous and frequently incomplete.

Abstract

From clinical healthcare to daily living, continuous sensor monitoring across multiple modalities has shown great promise for real-world intelligent decision-making but also faces various challenges. In this work, we introduce MAESTRO, a novel framework that overcomes key limitations of existing multimodal learning approaches: (1) reliance on a single primary modality for alignment, (2) pairwise modeling of modalities, and (3) assumption of complete modality observations. These limitations hinder the applicability of these approaches in real-world multimodal time-series settings, where primary modality priors are often unclear, the number of modalities can be large (making pairwise modeling impractical), and sensor failures often result in arbitrary missing observations. At its core, MAESTRO facilitates dynamic intra- and cross-modal interactions based on task relevance, and leverages symbolic tokenization and adaptive attention budgeting to construct long multimodal sequences, which are processed via sparse cross-modal attention. The resulting cross-modal tokens are routed through a sparse Mixture-of-Experts (MoE) mechanism, enabling black-box specialization under varying modality combinations. We evaluate MAESTRO against 10 baselines on four diverse datasets spanning three applications, and observe average relative improvements of 4% and 8% over the best existing multimodal and multivariate approaches, respectively, under complete observations. Under partial observations -- with up to 40% of missing modalities -- MAESTRO achieves an average 9% improvement. Further analysis also demonstrates the robustness and efficiency of MAESTRO's sparse, modality-aware design for learning from dynamic time series.

Figures (20)

• Figure 1: Illustration of traditional multivariate processing vs. our multimodal handling of sensor data, highlighting our method's superior performance and robustness.
• Figure 2: Overview of our approach, MAESTRO. Input data from arbitrary combinations of sensing modalities is tokenized using symbolic approximation, where a reserved symbol is used to denote missing modalities. A learnable attention budget gate to allocates modality-wise attention capacity for sparse-attention-based modality-specific encoders. The resulting modality-specific features are concatenated and combined with modality and positional embeddings, forming a long multimodal sequence, which is processed by a sparse cross-modal multihead-attention layer(s). The resulting tokens are routed through a Sparse Mixture-of-Experts module, enabling dynamic specialization under arbitrary observability conditions. Finally, a classifier maps the aggregated representation to task predictions.
• Figure 3: Reconstruction from symbolic tokenization of a PPG signal.
• Figure 4: Expert routing decisions across different input modality combinations, highlighting the implicit specialization behavior of the sparse MoE-layer.
• Figure 5: Comparative performance (Macro-F1 score) of MAESTRO against missingness-aware multimodal baselines and a modality-dropout-adapted Transformer shows its consistent superiority across varying missingness levels on (a) WESAD, (b) DaliaHAR, (c) DSADS, and (d) MIMIC-III.

Theorems & Definitions (2)

• Definition 3.1: Multimodal Time-series
• Corollary 3.2: Cross-Modal Relational Preservation with Bounded SAX Distortion