Online Multi-level Contrastive Representation Distillation for Cross-Subject fNIRS Emotion Recognition

TL;DR

This work proposes a novel cross-subject fNIRS emotion recognition method, called the Online Multi-level Contrastive Representation Distillation framework (OMCRD), a framework designed for mutual learning among multiple lightweight student networks, and facilitates knowledge transfer for interactions between student models.

Abstract

Utilizing functional near-infrared spectroscopy (fNIRS) signals for emotion recognition is a significant advancement in understanding human emotions. However, due to the lack of artificial intelligence data and algorithms in this field, current research faces the following challenges: 1) The portable wearable devices have higher requirements for lightweight models; 2) The objective differences of physiology and psychology among different subjects aggravate the difficulty of emotion recognition. To address these challenges, we propose a novel cross-subject fNIRS emotion recognition method, called the Online Multi-level Contrastive Representation Distillation framework (OMCRD). Specifically, OMCRD is a framework designed for mutual learning among multiple lightweight student networks. It utilizes multi-level fNIRS feature extractor for each sub-network and conducts multi-view sentimental mining using physiological signals. The proposed Inter-Subject Interaction Contrastive Representation (IS-ICR) facilitates knowledge transfer for interactions between student models, enhancing cross-subject emotion recognition performance. The optimal student network can be selected and deployed on a wearable device. Some experimental results demonstrate that OMCRD achieves state-of-the-art results in emotional perception and affective imagery tasks.

Figures (6)

• Figure 1: (a) Overview of the proposed OMCRD for fNIRS emotion recognition. The pictures of emotional stimulus sources are taken from spape2023nemo. (b) and (c) show two different types of multi-level fNIRS feature extractors.
• Figure 2: The architecture of the encoding branch unit in the Transformer extractor.
• Figure 3: Overview of the proposed IS-ICR. $f_a$ and $f_b$ represent two different sub-networks in the framework. $\boldsymbol{v}_m^i$ is the feature embedding vector of the input instance $\boldsymbol{x}_i$ obtained through the network $f_m$.
• Figure 4: The channel location of fNIRS. Orange circles are 10 transmitters, blue circles are 8 receivers, and green lines are 24 fNIRS channels.
• Figure 5: Ablation study results with varying numbers of peers. (a) and (b) show the results of OMCRD based on the CNN+LSTM extractor on the Empe and Afim tasks. (c) and (d) show the results based on the Transformer extractor. Dashed lines indicate baseline results