Can Brain Signals Reveal Inner Alignment with Human Languages?

TL;DR

This work investigates whether brain signals can reveal inner alignment with human languages by introducing MTAM, a Multimodal Transformer Alignment Model that aligns EEG and language representations via a dual-encoder with cross-domain alignment losses. It demonstrates state-of-the-art results on sentiment analysis and relation detection on ZuCo and K-EmoCon datasets, and provides interpretability analyses including feature distributions, alignment weights, and brain topographies. The study anchors the neurophysiological basis of language processing in EEG-language connectivity and offers visualizations and brain mappings to support interpretations. It contributes a first-of-its-kind framework for EEG-language alignment with publicly available code, enabling further exploration and applications in cognitive NLP.

Abstract

Brain Signals, such as Electroencephalography (EEG), and human languages have been widely explored independently for many downstream tasks, however, the connection between them has not been well explored. In this study, we explore the relationship and dependency between EEG and language. To study at the representation level, we introduced \textbf{MTAM}, a \textbf{M}ultimodal \textbf{T}ransformer \textbf{A}lignment \textbf{M}odel, to observe coordinated representations between the two modalities. We used various relationship alignment-seeking techniques, such as Canonical Correlation Analysis and Wasserstein Distance, as loss functions to transfigure features. On downstream applications, sentiment analysis and relation detection, we achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an F1-score improvement of 1.7% on K-EmoCon and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCo for relation detection. In addition, we provide interpretations of the performance improvement: (1) feature distribution shows the effectiveness of the alignment module for discovering and encoding the relationship between EEG and language; (2) alignment weights show the influence of different language semantics as well as EEG frequency features; (3) brain topographical maps provide an intuitive demonstration of the connectivity in the brain regions. Our code is available at \url{https://github.com/Jason-Qiu/EEG_Language_Alignment}.

Figures (11)

• Figure 1: The architecture of our model, where EEG and language features are coordinately explored by two encoders. The EEG encoder and language encoder are shown on the left and right, respectively. The cross-alignment module is used to explore the connectivity and relationship within two domains, while the transformed features are used for downstream tasks.
• Figure 2: Negative word-level alignment.
• Figure 3: Positive word-level alignment.
• Figure 4: Brain topologies.
• Figure 5: Three paradigms of EEG and language alignment.