Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning

Arian Khorasani, Théophile Demazure

TL;DR

The paper tackles the problem of fusing asynchronous EEG and peripheral physiology for affect modeling. It introduces Cross-Temporal Attention Fusion (CTAF), a self-supervised module that learns soft cross-temporal alignments via bidirectional cross-attention and builds a robust clip embedding $z_f$ through a mask-aware fusion mechanism and alignment-guided contrastive objectives, with optional weak supervision. On the K-EmoCon dataset with subject-wise LOOCV, CTAF yields stronger cross-modal alignment (larger cosine margins between matched and mismatched pairs) and improved cross-modal token retrieval within a one-second tolerance, while remaining competitive on discretized arousal/valence tasks with limited labels. These results establish CTAF as a practical, label-efficient approach for generalizable EEG–peripheral fusion under temporal asynchrony and provide diagnostic metrics explicitly tied to alignment quality.

Abstract

We study multimodal affect modeling when EEG and peripheral physiology are asynchronous, which most fusion methods ignore or handle with costly warping. We propose Cross-Temporal Attention Fusion (CTAF), a self-supervised module that learns soft bidirectional alignments between modalities and builds a robust clip embedding using time-aware cross attention, a lightweight fusion gate, and alignment-regularized contrastive objectives with optional weak supervision. On the K-EmoCon dataset, under leave-one-out cross-validation evaluation, CTAF yields higher cosine margins for matched pairs and better cross-modal token retrieval within one second, and it is competitive with the baseline on three-bin accuracy and macro-F1 while using few labels. Our contributions are a time-aware fusion mechanism that directly models correspondence, an alignment-driven self-supervised objective tailored to EEG and physiology, and an evaluation protocol that measures alignment quality itself. Our approach accounts for the coupling between the central and autonomic nervous systems in psychophysiological time series. These results indicate that CTAF is a strong step toward label-efficient, generalizable EEG-peripheral fusion under temporal asynchrony.

Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning

TL;DR

The paper tackles the problem of fusing asynchronous EEG and peripheral physiology for affect modeling. It introduces Cross-Temporal Attention Fusion (CTAF), a self-supervised module that learns soft cross-temporal alignments via bidirectional cross-attention and builds a robust clip embedding through a mask-aware fusion mechanism and alignment-guided contrastive objectives, with optional weak supervision. On the K-EmoCon dataset with subject-wise LOOCV, CTAF yields stronger cross-modal alignment (larger cosine margins between matched and mismatched pairs) and improved cross-modal token retrieval within a one-second tolerance, while remaining competitive on discretized arousal/valence tasks with limited labels. These results establish CTAF as a practical, label-efficient approach for generalizable EEG–peripheral fusion under temporal asynchrony and provide diagnostic metrics explicitly tied to alignment quality.

Abstract

We study multimodal affect modeling when EEG and peripheral physiology are asynchronous, which most fusion methods ignore or handle with costly warping. We propose Cross-Temporal Attention Fusion (CTAF), a self-supervised module that learns soft bidirectional alignments between modalities and builds a robust clip embedding using time-aware cross attention, a lightweight fusion gate, and alignment-regularized contrastive objectives with optional weak supervision. On the K-EmoCon dataset, under leave-one-out cross-validation evaluation, CTAF yields higher cosine margins for matched pairs and better cross-modal token retrieval within one second, and it is competitive with the baseline on three-bin accuracy and macro-F1 while using few labels. Our contributions are a time-aware fusion mechanism that directly models correspondence, an alignment-driven self-supervised objective tailored to EEG and physiology, and an evaluation protocol that measures alignment quality itself. Our approach accounts for the coupling between the central and autonomic nervous systems in psychophysiological time series. These results indicate that CTAF is a strong step toward label-efficient, generalizable EEG-peripheral fusion under temporal asynchrony.
Paper Structure (10 sections, 20 equations, 3 figures, 2 tables)

This paper contains 10 sections, 20 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Cross-Temporal Attention Fusion (CTAF)
  4. Preliminaries and Problem Setup
  5. CTAF Architecture
  6. Learning Objectives and Training Strategy
  7. Other Design Choices
  8. Empirical Evaluation
  9. Results
  10. Discussion and Conclusion

Figures (3)

  • Figure 1: Clip-level cosine similarity distributions for matched (cos_pos) vs. mismatched (cos_neg) EEG–physiology pairs, aggregated over LOOCV folds. Larger separation indicates stronger cross-modal alignment; boxes show median and IQR across participants (PIDs).
  • Figure 2: Per-participant (PID) matched cosine (cos_pos) WITH time (CTAF) vs. NO time ablation. Points above the diagonal indicate improved alignment when using CTAF’s time-aware fusion.
  • Figure 3: Cross-modal token-retrieval accuracy within tolerance $\tau$ seconds for EEG$\rightarrow$Phys and Phys$\rightarrow$EEG. Bars compare WITH time (CTAF) to NO time ablation; error bars show 95% bootstrap CIs across participants.