Leveraging Vision Transformers for Enhanced Classification of Emotions using ECG Signals

TL;DR

ECG-based emotion recognition remains challenging due to limited models capturing long-range dependencies in time-frequency signals. This work introduces the Enhanced ECG Signal Vision Transformer (ES-ViT), which fuses a CNN-derived embedding with squeeze-and-excitation recalibration and integrates a global image embedding into each Transformer layer, operating on RGB ECG images encoded via Continuous Wavelet Transform and Power Spectral Density. Evaluations on YAAD and DREAMER show ES-ViT achieving state-of-the-art performance across emotion, arousal, valence, and dominance metrics, surpassing ResNet, MobileNet, and other baselines. The results indicate ES-ViT's potential for accurate, scalable emotion monitoring from ECG in healthcare, HCI, and wearable-enabled applications.

Abstract

Biomedical signals provide insights into various conditions affecting the human body. Beyond diagnostic capabilities, these signals offer a deeper understanding of how specific organs respond to an individual's emotions and feelings. For instance, ECG data can reveal changes in heart rate variability linked to emotional arousal, stress levels, and autonomic nervous system activity. This data offers a window into the physiological basis of our emotional states. Recent advancements in the field diverge from conventional approaches by leveraging the power of advanced transformer architectures, which surpass traditional machine learning and deep learning methods. We begin by assessing the effectiveness of the Vision Transformer (ViT), a forefront model in image classification, for identifying emotions in imaged ECGs. Following this, we present and evaluate an improved version of ViT, integrating both CNN and SE blocks, aiming to bolster performance on imaged ECGs associated with emotion detection. Our method unfolds in two critical phases: first, we apply advanced preprocessing techniques for signal purification and converting signals into interpretable images using continuous wavelet transform and power spectral density analysis; second, we unveil a performance-boosted vision transformer architecture, cleverly enhanced with convolutional neural network components, to adeptly tackle the challenges of emotion recognition. Our methodology's robustness and innovation were thoroughly tested using ECG data from the YAAD and DREAMER datasets, leading to remarkable outcomes. For the YAAD dataset, our approach outperformed existing state-of-the-art methods in classifying seven unique emotional states, as well as in valence and arousal classification. Similarly, in the DREAMER dataset, our method excelled in distinguishing between valence, arousal and dominance, surpassing current leading techniques.

Figures (2)

• Figure 1: Proposed architecture for ECG data classification using vision transformers.
• Figure 2: Signal Processing steps.