Attention Isn't All You Need for Emotion Recognition:Domain Features Outperform Transformers on the EAV Dataset

TL;DR

This work addresses multimodal emotion recognition on a small dataset (EAV) and systematically compares baseline transformers (M1), factorized attention models (M2), and minimal CNN improvements (M3). It finds that domain-aware feature engineering and careful bug fixes outperform architectural complexity: M2 underperforms by $5$–$13$ percentage points, while M3 achieves state-of-the-art results via delta MFCCs for audio, band-power/alpha-asymmetry for EEG, and delta features for vision, with vision pretraining domain proving especially impactful. The study highlights the importance of matching model capacity to data, preserving pretrained representations, and leveraging neuroscience-informed features, suggesting that complex inductive biases require more data to pay off. Practically, these results advocate prioritizing domain knowledge and bug-free baselines before pursuing architectural complexity, and point to the potential of factorized attention in larger emotion datasets.

Abstract

We present a systematic study of multimodal emotion recognition using the EAV dataset, investigating whether complex attention mechanisms improve performance on small datasets. We implement three model categories: baseline transformers (M1), novel factorized attention mechanisms (M2), and improved CNN baselines (M3). Our experiments show that sophisticated attention mechanisms consistently underperform on small datasets. M2 models achieved 5 to 13 percentage points below baselines due to overfitting and destruction of pretrained features. In contrast, simple domain-appropriate modifications proved effective: adding delta MFCCs to the audio CNN improved accuracy from 61.9\% to \textbf{65.56\%} (+3.66pp), while frequency-domain features for EEG achieved \textbf{67.62\%} (+7.62pp over the paper baseline). Our vision transformer baseline (M1) reached \textbf{75.30\%}, exceeding the paper's ViViT result (74.5\%) through domain-specific pretraining, and vision delta features achieved \textbf{72.68\%} (+1.28pp over the paper CNN). These findings demonstrate that for small-scale emotion recognition, domain knowledge and proper implementation outperform architectural complexity.

Figures (2)

• Figure 1: Architecture diagrams for M2: Factorized Attention Mechanisms. Left: EEG Tri-Stream Transformer with spatial, temporal, and asymmetry attention streams. Center: Audio Dual Attention with temporal-frequency factorization over AST features. Right: Vision Space-Time Attention with factorized spatial and temporal attention over ViT embeddings. All architectures include skip connections to preserve pretrained features.
• Figure 2: Architecture diagrams for M3: CNN Improvements. Left: EEG Band Power Features with frequency-domain feature extraction (band power, differential entropy, and alpha asymmetry) feeding a simple MLP. Center: Vision CNN with ResNet50 backbone and fixed squeeze-excitation block. Right: Audio CNN with delta MFCC features added to the original feature set.