Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking

Shohreh Deldari, Dimitris Spathis, Mohammad Malekzadeh, Fahim Kawsar, Flora Salim, Akhil Mathur

TL;DR

CroSSL proposes a cross-modal self-supervised framework for multimodal time-series that uses modality-specific encoders, a cross-modal aggregator, and latent masking to learn global embeddings without requiring labeled data or negative-pair sampling. Grounded in an information-theoretic view and a VICReg-like loss, it enforces cross-modal sharing while discarding modality-specific details, and it demonstrates strong performance and missing-data robustness on HAR and biosignal datasets. The work shows substantial improvements over state-of-the-art SSL and supervised baselines, with notable gains from spatial masking and high label-efficiency, ultimately enabling robust cross-modal representations for edge-friendly deployment. Open-source code is provided, reinforcing practical impact in health monitoring and activity recognition tasks.

Abstract

Limited availability of labeled data for machine learning on multimodal time-series extensively hampers progress in the field. Self-supervised learning (SSL) is a promising approach to learning data representations without relying on labels. However, existing SSL methods require expensive computations of negative pairs and are typically designed for single modalities, which limits their versatility. We introduce CroSSL (Cross-modal SSL), which puts forward two novel concepts: masking intermediate embeddings produced by modality-specific encoders, and their aggregation into a global embedding through a cross-modal aggregator that can be fed to down-stream classifiers. CroSSL allows for handling missing modalities and end-to-end cross-modal learning without requiring prior data preprocessing for handling missing inputs or negative-pair sampling for contrastive learning. We evaluate our method on a wide range of data, including motion sensors such as accelerometers or gyroscopes and biosignals (heart rate, electroencephalograms, electromyograms, electrooculograms, and electrodermal) to investigate the impact of masking ratios and masking strategies for various data types and the robustness of the learned representations to missing data. Overall, CroSSL outperforms previous SSL and supervised benchmarks using minimal labeled data, and also sheds light on how latent masking can improve cross-modal learning. Our code is open-sourced at https://github.com/dr-bell/CroSSL.

CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking

TL;DR

CroSSL proposes a cross-modal self-supervised framework for multimodal time-series that uses modality-specific encoders, a cross-modal aggregator, and latent masking to learn global embeddings without requiring labeled data or negative-pair sampling. Grounded in an information-theoretic view and a VICReg-like loss, it enforces cross-modal sharing while discarding modality-specific details, and it demonstrates strong performance and missing-data robustness on HAR and biosignal datasets. The work shows substantial improvements over state-of-the-art SSL and supervised baselines, with notable gains from spatial masking and high label-efficiency, ultimately enabling robust cross-modal representations for edge-friendly deployment. Open-source code is provided, reinforcing practical impact in health monitoring and activity recognition tasks.

Abstract

Limited availability of labeled data for machine learning on multimodal time-series extensively hampers progress in the field. Self-supervised learning (SSL) is a promising approach to learning data representations without relying on labels. However, existing SSL methods require expensive computations of negative pairs and are typically designed for single modalities, which limits their versatility. We introduce CroSSL (Cross-modal SSL), which puts forward two novel concepts: masking intermediate embeddings produced by modality-specific encoders, and their aggregation into a global embedding through a cross-modal aggregator that can be fed to down-stream classifiers. CroSSL allows for handling missing modalities and end-to-end cross-modal learning without requiring prior data preprocessing for handling missing inputs or negative-pair sampling for contrastive learning. We evaluate our method on a wide range of data, including motion sensors such as accelerometers or gyroscopes and biosignals (heart rate, electroencephalograms, electromyograms, electrooculograms, and electrodermal) to investigate the impact of masking ratios and masking strategies for various data types and the robustness of the learned representations to missing data. Overall, CroSSL outperforms previous SSL and supervised benchmarks using minimal labeled data, and also sheds light on how latent masking can improve cross-modal learning. Our code is open-sourced at https://github.com/dr-bell/CroSSL.
Paper Structure (15 sections, 6 equations, 4 figures, 4 tables)

This paper contains 15 sections, 6 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method: CroSSL
  4. Problem Formulation
  5. Theoretical Motivations
  6. Evaluation
  7. Datasets
  8. Experiments Setup
  9. Baselines
  10. Comparison at One Glance
  11. Robustness Against Missing Data
  12. Optimal Masking Strategies
  13. Label-efficiency
  14. Discussion and Limitations
  15. Conclusion

Figures (4)

  • Figure 1: The overview of the proposed architecture.
  • Figure 2: Overview of the regularization-based objective function within the CroSSL architecture.
  • Figure 3: Comparing random (a,b,c) and spatial (d,e,f) latent masking across the three datasets for different masking rates.
  • Figure 4: Comparing the efficiency of fixed and fine-tuned CroSSL setups against fully supervised model in low-labeled data