Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

E^2-LLM: Bridging Neural Signals and Interpretable Affective Analysis

Fei Ma, Han Lin, Yifan Xie, Hongwei Ren, Xiaoyu Shen, Wenbo Ding, Qi Tian

TL;DR

This work tackles EEG-based emotion recognition by addressing variability and data scarcity through a novel framework, E^2-LLM, which bridges EEG signals to interpretable language-based emotion analysis. It introduces a hierarchical EEG encoder, a learnable Projector, and a Qwen3-based LLM, connected via a multi-stage training curriculum and instructed chain-of-thought reasoning to produce natural-language emotional explanations. Evaluation on SEED-VII across seven emotion categories demonstrates that larger LLM backbones improve classification accuracy and zero-shot reasoning, with detailed ablations underscoring the necessity of staged training and CoT prompts. The study establishes a new paradigm that combines physiological signals with scalable LLM reasoning, advancing both recognition performance and interpretable affective understanding, while highlighting practical considerations for dataset diversity, computational resources, and ethical deployment.

Abstract

Emotion recognition from electroencephalography (EEG) signals remains challenging due to high inter-subject variability, limited labeled data, and the lack of interpretable reasoning in existing approaches. While recent multimodal large language models (MLLMs) have advanced emotion analysis, they have not been adapted to handle the unique spatiotemporal characteristics of neural signals. We present E^2-LLM (EEG-to-Emotion Large Language Model), the first MLLM framework for interpretable emotion analysis from EEG. E^2-LLM integrates a pretrained EEG encoder with Qwen-based LLMs through learnable projection layers, employing a multi-stage training pipeline that encompasses emotion-discriminative pretraining, cross-modal alignment, and instruction tuning with chain-of-thought reasoning. We design a comprehensive evaluation protocol covering basic emotion prediction, multi-task reasoning, and zero-shot scenario understanding. Experiments on the dataset across seven emotion categories demonstrate that E^2-LLM achieves excellent performance on emotion classification, with larger variants showing enhanced reliability and superior zero-shot generalization to complex reasoning scenarios. Our work establishes a new paradigm combining physiological signals with LLM reasoning capabilities, showing that model scaling improves both recognition accuracy and interpretable emotional understanding in affective computing.

E^2-LLM: Bridging Neural Signals and Interpretable Affective Analysis

TL;DR

This work tackles EEG-based emotion recognition by addressing variability and data scarcity through a novel framework, E^2-LLM, which bridges EEG signals to interpretable language-based emotion analysis. It introduces a hierarchical EEG encoder, a learnable Projector, and a Qwen3-based LLM, connected via a multi-stage training curriculum and instructed chain-of-thought reasoning to produce natural-language emotional explanations. Evaluation on SEED-VII across seven emotion categories demonstrates that larger LLM backbones improve classification accuracy and zero-shot reasoning, with detailed ablations underscoring the necessity of staged training and CoT prompts. The study establishes a new paradigm that combines physiological signals with scalable LLM reasoning, advancing both recognition performance and interpretable affective understanding, while highlighting practical considerations for dataset diversity, computational resources, and ethical deployment.

Abstract

Emotion recognition from electroencephalography (EEG) signals remains challenging due to high inter-subject variability, limited labeled data, and the lack of interpretable reasoning in existing approaches. While recent multimodal large language models (MLLMs) have advanced emotion analysis, they have not been adapted to handle the unique spatiotemporal characteristics of neural signals. We present E^2-LLM (EEG-to-Emotion Large Language Model), the first MLLM framework for interpretable emotion analysis from EEG. E^2-LLM integrates a pretrained EEG encoder with Qwen-based LLMs through learnable projection layers, employing a multi-stage training pipeline that encompasses emotion-discriminative pretraining, cross-modal alignment, and instruction tuning with chain-of-thought reasoning. We design a comprehensive evaluation protocol covering basic emotion prediction, multi-task reasoning, and zero-shot scenario understanding. Experiments on the dataset across seven emotion categories demonstrate that E^2-LLM achieves excellent performance on emotion classification, with larger variants showing enhanced reliability and superior zero-shot generalization to complex reasoning scenarios. Our work establishes a new paradigm combining physiological signals with LLM reasoning capabilities, showing that model scaling improves both recognition accuracy and interpretable emotional understanding in affective computing.
Paper Structure (29 sections, 8 equations, 3 figures, 4 tables)

This paper contains 29 sections, 8 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based EEG Analysis
  4. LLM-based Emotion Analysis
  5. Multimodal Large Language Models
  6. Method
  7. Data Construction
  8. E$^2$-LLM
  9. Overview
  10. EEG Signal Processing
  11. Hierarchical Encoding
  12. Multimodal Alignment
  13. Input Formulation
  14. Multi-stage Training Strategy
  15. Stage 1: EEG Encoder Training
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Overview of the E²-LLM framework and training pipeline. (a) E$^2$-LLM framework: EEG signals are processed by an EEG Encoder, whose representations are mapped to the embedding space of a Qwen3-based LLM via a Projector. Special tokens $\texttt{<eeg\_start>}$ and $\texttt{<eeg\_end>}$ demarcate EEG segments within the input sequence, enabling the LLM to generate interpretive emotional analysis. (b) Multi-Stage Training Strategy: Stage 1 trains the EEG Encoder with a classification objective for emotion recognition. Stage 2 freezes the encoder and trains the Projector to align EEG representations with LLM embeddings. Stage 3 jointly fine-tunes the Projector and LLM for generating natural language emotional analysis reports.
  • Figure 2: Illustrative examples from the proposed E$^2$-LLM across five distinct tasks. The qualitative results demonstrate that E$^2$-LLM can generate reasonable outputs.
  • Figure 3: Ablation analysis on the IED task across different model scales.