Neural Tracking of Sustained Attention, Attention Switching, and Natural Conversation in Audiovisual Environments using Mobile EEG

TL;DR

This work addresses neural tracking of selective auditory attention in naturalistic audiovisual environments using a mobile EEG system. It introduces an AV three-task dataset (SustAC, SwitAC, ConvAC) with 24 participants and 44 scalp plus cEEGrid electrodes, and applies TRF-based forward/backward models to quantify attention decoding from EEG. The results show robust attended-vs-ignored differences in P2 components and reconstruction/classification performance across tasks, with generalization of models across conditions, though cEEGrid performance is weaker. These findings support real-world attention-tracking applications and inform the design of AV attention paradigms for wearable neurotech.

Abstract

Everyday communication is dynamic and multisensory, often involving shifting attention, overlapping speech and visual cues. Yet, most neural attention tracking studies are still limited to highly controlled lab settings, using clean, often audio-only stimuli and requiring sustained attention to a single talker. This work addresses that gap by introducing a novel dataset from 24 normal-hearing participants. We used a mobile electroencephalography (EEG) system (44 scalp electrodes and 20 cEEGrid electrodes) in an audiovisual (AV) paradigm with three conditions: sustained attention to a single talker in a two-talker environment, attention switching between two talkers, and unscripted two-talker conversations with a competing single talker. Analysis included temporal response functions (TRFs) modeling, optimal lag analysis, selective attention classification with decision windows ranging from 1.1s to 35s, and comparisons of TRFs for attention to AV conversations versus side audio-only talkers. Key findings show significant differences in the attention-related P2-peak between attended and ignored speech across conditions for scalp EEG. No significant change in performance between switching and sustained attention suggests robustness for attention switches. Optimal lag analysis revealed narrower peak for conversation compared to single-talker AV stimuli, reflecting the additional complexity of multi-talker processing. Classification of selective attention was consistently above chance (55-70% accuracy) for scalp EEG, while cEEGrid data yielded lower correlations, highlighting the need for further methodological improvements. These results demonstrate that mobile EEG can reliably track selective attention in dynamic, multisensory listening scenarios and provide guidance for designing future AV paradigms and real-world attention tracking applications.

Figures (9)

• Figure 1: Experiment paradigm and condition designs. Top panels (a), (b), and (c) show the experimental setup of SustAC, SwitAC and ConvAC, respectively. The setups for SustAC (a) and SwitAC (b) are identical, with attention directed to one out of the two frontal AV speakers. In (c), attention is directed either to the two frontal AV speakers engaged in conversation or to the side single speaker. Below each setup, the instructed attentional focus over the course of a trial is shown: attention is sustained on a single talker for SustAC and ConvAC, whereas SwitAC includes two switches between speakers.
• Figure 2: The EEG setup with common mode sense (CMS) and driven right leg (DRL) electrodes, and subsequent preprocessing of stimuli (blue) and EEG data (orange) are shown as flowcharts. Forward and backward temporal response functions (TRF) models are then fitted using cross-validation (CV) (red). The forward model is analyzed both qualitatively and quantitatively by inspecting TRF waveforms, while the backward model is analyzed quantitatively via correlation-based metrics, including reconstruction accuracy and classification performance (green).
• Figure 3: Behavioral analysis: Answer rate (left), self-rated listening difficulty (middle), and self-rated understanding difficulty (right) for the three conditions, SustAC, SwitAC and ConvAC. Colored dots show individual subject averages; black markers indicate the group mean. Ratings are on a $1-7$ scale. Significance was corrected for multiple comparisons using the Benjamini-Hochberg procedure benjaminihochberg.
• Figure 4: TRF peak analysis. TRFs for our three conditions SustAC(A-D), SwitAC(E-H), and ConvAC(I-L), shown for attended (blue) and ignored (orange), with random control speech (grey). The left two and right two columns show TRFs for the acoustic envelope and the acoustic onset, respectively. The topographic plots show spatial patterns at observed peaks of interest, where red indicates positive amplitude and blue negative amplitude. All presented plots are for the scalp EEG sensors, with the thicker lines showing the TRF channel-average, as indicated by the cap layout at the bottom.
• Figure 5: TRF cluster analysis: statistical cluster analysis on the differences between attended and ignored TRFs presented in Figure \ref{['fig:trf']}. AB is the result from Figures \ref{['fig:trf']}A and \ref{['fig:trf']}B, CD from \ref{['fig:trf']}C and \ref{['fig:trf']}D, etc. The analysis used an independent samples t-test in Eelbrain, where $p<0.05$ clusters are shown in blue.