Multimodal Functional Maximum Correlation for Emotion Recognition

TL;DR

MFMC introduces a principled self-supervised framework to learn emotion representations from multiple physiological modalities by maximizing higher-order dependence via a Dual Total Correlation bound and a FMCA-based trace surrogate. It avoids positive/negative sampling and heavy augmentations, directly capturing joint interactions across modalities such as EEG, ECG, EDA, and SKT. Across DEAP, CEAP-360VR, and MAHNOB-HCI, MFMC achieves state-of-the-art or competitive results under both subject-dependent and subject-independent protocols, often approaching supervised baselines in cross-subject scenarios. The approach demonstrates strong generalization to inter-subject variability and offers practical deployment benefits, with code available for reproducibility.

Abstract

Emotional states manifest as coordinated yet heterogeneous physiological responses across central and autonomic systems, posing a fundamental challenge for multimodal representation learning in affective computing. Learning such joint dynamics is further complicated by the scarcity and subjectivity of affective annotations, which motivates the use of self-supervised learning (SSL). However, most existing SSL approaches rely on pairwise alignment objectives, which are insufficient to characterize dependencies among more than two modalities and fail to capture higher-order interactions arising from coordinated brain and autonomic responses. To address this limitation, we propose Multimodal Functional Maximum Correlation (MFMC), a principled SSL framework that maximizes higher-order multimodal dependence through a Dual Total Correlation (DTC) objective. By deriving a tight sandwich bound and optimizing it using a functional maximum correlation analysis (FMCA) based trace surrogate, MFMC captures joint multimodal interactions directly, without relying on pairwise contrastive losses. Experiments on three public affective computing benchmarks demonstrate that MFMC consistently achieves state-of-the-art or competitive performance under both subject-dependent and subject-independent evaluation protocols, highlighting its robustness to inter-subject variability. In particular, MFMC improves subject-dependent accuracy on CEAP-360VR from 78.9% to 86.8%, and subject-independent accuracy from 27.5% to 33.1% using the EDA signal alone. Moreover, MFMC remains within 0.8 percentage points of the best-performing method on the most challenging EEG subject-independent split of MAHNOB-HCI. Our code is available at https://github.com/DY9910/MFMC.

Figures (6)

• Figure 1: TC and DTC on three variables $X_1, X_2, X_3$. In each case, the quantity is represented by the total number of block areas of the diagram. TC counts the triple-overlapped shaded area (a.k.a., interaction information mcgill1954multivariate) twice, whereas DTC counts this area just once.
• Figure 2: Diagram illustrating the maximization total dependence between {EEG, EOG} and skin temperature by decomposing the density ratio $\frac{p(X_1, X_2, X_3)}{p(X_1, X_2)p(X_3)}$. Networks $f_\theta$ ($C_1$ channels) and $g_\phi$ ($C_2$ channels) are applied to $X_1$ and $X_2$, respectively, to extract EEG and EOG embeddings $e_1$ and $e_2$, which are then fused via a fusion network to obtain the joint embedding $e_{12}$. Similarly, $h_\psi$ encodes $X_3$ ($C_3$ channels) into $e_{3}$. Finally, $e_{12}$ and $e_{3}$ are used to optimize the trace loss $\mathrm{tr}(R_{12}^{-1} P_{12,3} R_3^{-1} P_{12,3}^\top)$, where $R$ and $P$ denote the sample auto- and cross-covariance matrices, respectively. The optimization converges to $- \sum_{i=1}^K \sigma_i$.
• Figure 3: Emotion-discrimination performance versus window length. Both macro-$F_{1}$ (blue) and accuracy (green) peak at 10 s, remain flat up to 14 s, and deteriorate beyond that. Very short windows ($<3$ s) perform markedly worse.
• Figure 4: Learnable attention weights for physiological signal selection across CEAP-360VR, DEAP, and MAHNOB-HCI datasets. Green bars indicate retained modalities, red bars indicate discarded modalities based on learned importance scores.
• Figure 5: Validation curves on the DEAP subject-dependent split. MFMC converges smoothly and attains the highest accuracy. High-order InfoNCE plateaus early; LogDet diverges after 13 k iterations.

Theorems & Definitions (8)

• Lemma 1: hu2022normalized
• Theorem 1
• proof
• Theorem 2
• Definition 1: Generalized definition of TSD
• Lemma 2: First-order approximation
• Lemma 3: austin2018multi
• Lemma 4: austin2018multi