Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals

Tomoyoshi Kimura, Xinlin Li, Osama Hanna, Yatong Chen, Yizhuo Chen, Denizhan Kara, Tianshi Wang, Jinyang Li, Xiaomin Ouyang, Shengzhong Liu, Mani Srivastava, Suhas Diggavi, Tarek Abdelzaher

TL;DR

InfoMAE addresses the data-efficiency challenge of cross-modal learning for time-series sensing in IoT by separating unimodal pretraining from cross-modal alignment and grounding alignment in an information-theoretic factorization into a shared representation $U$ and private representations $V$. It combines distribution-level alignment with reconstruction, instance-level contrast, and temporal locality to robustly fuse heterogeneous modalities using only limited synchronized data, while leveraging abundant unimodal data. Empirical results across MOD and HAR demonstrate strong improvements in multimodal tasks (often >60% downstream gains) and notable unimodal gains, with substantial data-efficiency advantages when multimodal pairs are scarce. The framework also remains competitive when abundant synchronized data are available, highlighting its versatility for both low-data and standard multimodal pretraining regimes in real-world IoT sensing scenarios.

Abstract

Standard multimodal self-supervised learning (SSL) algorithms regard cross-modal synchronization as implicit supervisory labels during pretraining, thus posing high requirements on the scale and quality of multimodal samples. These constraints significantly limit the performance of sensing intelligence in IoT applications, as the heterogeneity and the non-interpretability of time-series signals result in abundant unimodal data but scarce high-quality multimodal pairs. This paper proposes InfoMAE, a cross-modal alignment framework that tackles the challenge of multimodal pair efficiency under the SSL setting by facilitating efficient cross-modal alignment of pretrained unimodal representations. InfoMAE achieves \textit{efficient cross-modal alignment} with \textit{limited data pairs} through a novel information theory-inspired formulation that simultaneously addresses distribution-level and instance-level alignment. Extensive experiments on two real-world IoT applications are performed to evaluate InfoMAE's pairing efficiency to bridge pretrained unimodal models into a cohesive joint multimodal model. InfoMAE enhances downstream multimodal tasks by over 60% with significantly improved multimodal pairing efficiency. It also improves unimodal task accuracy by an average of 22%.

InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals

TL;DR

InfoMAE addresses the data-efficiency challenge of cross-modal learning for time-series sensing in IoT by separating unimodal pretraining from cross-modal alignment and grounding alignment in an information-theoretic factorization into a shared representation and private representations . It combines distribution-level alignment with reconstruction, instance-level contrast, and temporal locality to robustly fuse heterogeneous modalities using only limited synchronized data, while leveraging abundant unimodal data. Empirical results across MOD and HAR demonstrate strong improvements in multimodal tasks (often >60% downstream gains) and notable unimodal gains, with substantial data-efficiency advantages when multimodal pairs are scarce. The framework also remains competitive when abundant synchronized data are available, highlighting its versatility for both low-data and standard multimodal pretraining regimes in real-world IoT sensing scenarios.

Abstract

Standard multimodal self-supervised learning (SSL) algorithms regard cross-modal synchronization as implicit supervisory labels during pretraining, thus posing high requirements on the scale and quality of multimodal samples. These constraints significantly limit the performance of sensing intelligence in IoT applications, as the heterogeneity and the non-interpretability of time-series signals result in abundant unimodal data but scarce high-quality multimodal pairs. This paper proposes InfoMAE, a cross-modal alignment framework that tackles the challenge of multimodal pair efficiency under the SSL setting by facilitating efficient cross-modal alignment of pretrained unimodal representations. InfoMAE achieves \textit{efficient cross-modal alignment} with \textit{limited data pairs} through a novel information theory-inspired formulation that simultaneously addresses distribution-level and instance-level alignment. Extensive experiments on two real-world IoT applications are performed to evaluate InfoMAE's pairing efficiency to bridge pretrained unimodal models into a cohesive joint multimodal model. InfoMAE enhances downstream multimodal tasks by over 60% with significantly improved multimodal pairing efficiency. It also improves unimodal task accuracy by an average of 22%.

Paper Structure

This paper contains 44 sections, 1 theorem, 16 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Analysis of Cross-Modal Alignment
  3. Notation
  4. Problem Definition
  5. Factorization & Distributional Alignment
  6. Connection between Factorization and Cross-modal Alignment
  7. Distributional Alignment through Information-theory based Factorization
  8. InfoMAE
  9. Unimodal Pretraining
  10. Distribution-level Alignment
  11. Shared Representation
  12. Private Representation
  13. Self Reconstruction
  14. Instance-level Alignment
  15. Temporal Locality
  16. ...and 29 more sections

Key Result

proposition 1

For random variables $X_1$, $X_2$, if $U=s_1(X_1) = s_2(X_2)$, and there exists $W = g_1(X_1) = g_2(X_2)$ such that $X_1\perp \!\!\! \perp X_2\mid W$, then the following two statements are equivalent.

Figures (7)

  • Figure 1: Comparison of supervised learning, self-supervised learning, and pair-efficient self-supervised learning.
  • Figure 2: An illustration of instance-level vs. distribution-level Cross-Modal Alignment
  • Figure 3: Overview of InfoMAE's alignment in the information space. InfoMAE adopts an information theory-inspired objective to align the factorized representations. Best viewed in color.
  • Figure 4: Key learning objectives of InfoMAE.
  • Figure 5: Unimodal linear probing accuracy of MOD with and without cross-modal alignment.
  • ...and 2 more figures

Theorems & Definitions (4)

  • definition 1
  • definition 2
  • definition 3
  • proposition 1