Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition

Abstract

Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states. Prior methods face two main limitations: mechanically relying on independent unimodal performance, thereby missing genuine complementary contributions, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks. As inconsistent information density across heterogeneous modalities hinders inter-modal feature mining, we propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration. Specifically, we orthogonally decompose each modality's features into high, mid, and low-frequency components. Building upon this band-level routing, we design a modality-level router with a dual-path mechanism for fine-grained cross-band selection and cross-modal fusion. To mitigate shortcut learning from dominant modalities, we propose the Marginal Complementarity Module (MCM) to quantify performance loss when removing each modality via bi-modal comparison. The resulting complementarity distribution provides soft supervision, guiding the router to focus on modalities contributing unique information gains. Extensive experiments show our method achieves superior performance on the CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMSv2, and MIntRec benchmarks.

Figures (6)

• Figure 1: Two fundamental limitations of existing MER methods.(a) Coarse-grained fusion: treating each modality's temporal features as a homogeneous signal discards fine-grained emotional cues in specific frequency bands. (b) Shortcut learning: performance-based weighting over-relies on the dominant (text) modality, whereas visual and acoustic modalities carry the genuinely complementary cues. Complementarity-based weighting correctly amplifies these contributions.
• Figure 2: The overall architecture of Atsuko.(1)Spectral-Band Routing: Multi-modal temporal inputs are orthogonally decomposed into distinct frequency bands via graph Laplacian and dynamically weighted to yield band-enhanced features $\tilde{X}_m$. (2)Modality-Level Routing: A dual-path mechanism (pre- and post-attention) assigns dynamic fusion weights $w_m$ to the deep semantic features. (3)Marginal Complementarity Module (MCM): Bimodal contrastive branches quantify each modality's marginal contribution, providing soft supervision to guide the modality router toward complementary information gains, augmented by feature distillation.
• Figure 3: Hyperparameter sensitivity analysis on CMU-MOSI and CMU-MOSEI (Acc-7, %). Each sub-figure sweeps one hyperparameter while holding all others at their default values. The star marker ($\bigstar$) denotes the default configuration adopted in all experiments. Blue lines: MOSI (left $y$-axis); red lines: MOSEI (right $y$-axis).
• Figure 4: t-SNE visualization of feature distributions on MOSI test set. (a) Baseline (w/o SBN & MCM), (b) w/o MCM, (c) w/o SBN, and (d) Atsuko (Full). Different colors represent varying sentiment intensities. The proposed method shows clearer cluster margins and better separability.
• Figure 5: Visualization of modality weights predicted by the router on randomly selected CMU-MOSI test samples. Gray bars denote the baseline model without MCM, and colored bars denote the MCM-enhanced model.