Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals

TL;DR

This work tackles robust multimodal pretraining for biosignals under distributional shifts between training and deployment. It introduces bioFAME, a frequency-aware transformer that uses a fixed-size Fourier-based operator to learn representations in frequency space, paired with frequency-maintain latent-space masked autoencoding and a channel-independent design to handle varying modality configurations. Across unimodal and multimodal transfer tasks, bioFAME achieves a notable average improvement of $5.5\%$ over previous state-of-the-art and demonstrates resilience to modality dropout and substitutions, highlighting practical applicability in real-world settings. The approach offers a flexible, efficient path to leveraging multimodal biosignals for generalizable physiological state understanding, with future work focusing on interpretability of learned frequency filters and scaling to high-dimensional sensor arrays.

Abstract

Leveraging multimodal information from biosignals is vital for building a comprehensive representation of people's physical and mental states. However, multimodal biosignals often exhibit substantial distributional shifts between pretraining and inference datasets, stemming from changes in task specification or variations in modality compositions. To achieve effective pretraining in the presence of potential distributional shifts, we propose a frequency-aware masked autoencoder ($\texttt{bio}$FAME) that learns to parameterize the representation of biosignals in the frequency space. $\texttt{bio}$FAME incorporates a frequency-aware transformer, which leverages a fixed-size Fourier-based operator for global token mixing, independent of the length and sampling rate of inputs. To maintain the frequency components within each input channel, we further employ a frequency-maintain pretraining strategy that performs masked autoencoding in the latent space. The resulting architecture effectively utilizes multimodal information during pretraining, and can be seamlessly adapted to diverse tasks and modalities at test time, regardless of input size and order. We evaluated our approach on a diverse set of transfer experiments on unimodal time series, achieving an average of $\uparrow$5.5% improvement in classification accuracy over the previous state-of-the-art. Furthermore, we demonstrated that our architecture is robust in modality mismatch scenarios, including unpredicted modality dropout or substitution, proving its practical utility in real-world applications. Code is available at https://github.com/apple/ml-famae .

Figures (4)

• Figure 1: Motivation of our approach. (A) In multimodal biosignal systems, there exists substantial distributional shifts between the pretraining and inference datasets. (B) The distributional shifts often cause the shifts of representation in time-space, which can affect the model's generalization ability within modality and across modalities. (C) In the meantime, the representation in frequency-space typically would contain similar frequency components within modality, leading to more stable combinations in multimodal scenarios.
• Figure 2: Overview. (A) Previous approaches perform masking in the time domain, which causes shifts in the frequency components. Also, the encoders are unaware of the frequency information in time series. (B) To address the issues, we propose bioFAME , which (i) builds frequency awareness by directly learning frequency filters in the representation space, and (ii) performs masked autoencoding in the latent space to maintain frequency information during pretraining. (C) We implement bioFAME in the multimodal pretraining scheme, where the frequency-aware encoder ($\operatorname{FA-Enc}(\cdot)$) processes signals in a channel-independent manner, and extracts representations with multi-head filter layer with fixed-size Fourier operators. The frequency-maintain pretraining strategy further performs masked autoencoding in the latent space with separate reconstruction to guide the effective mixing of multimodal information.
• Figure 3: Multimodal evaluation results. (A) Two modality mismatch scenarios are considered: Modality substitution and modality dropout. (B) When a modality is swapped with another available one, or (C) when modalities are dropped out at test time, our model gives lower performance degradation when comparing to a robust baseline. (D) By visualizing the attention weights across modalities, we can understand how modalities are associated with each other.
• Figure 4: (A) We examine the performance of bioFAME under low-data regime with and without pretraining. (B) We examine how the MaxPool operator and Query operator would perform with different amounts of filters.