ListenNet: A Lightweight Spatio-Temporal Enhancement Nested Network for Auditory Attention Detection

TL;DR

ListenNet addresses EEG-based auditory attention detection by jointly modeling spatio-temporal dependencies with a lightweight architecture. It introduces a Spatio-temporal Dependency Encoder, Multi-scale Temporal Enhancement, and Cross-Nested Attention to capture dynamic cross-channel and multi-scale temporal patterns while maintaining a small parameter count. The approach achieves state-of-the-art results in subject-dependent and more challenging subject-independent settings across three public datasets, with substantial reductions in model size and computation. These advances enable robust, real-time AAD suitable for low-power devices and real-world hearing-assistive systems.

Abstract

Auditory attention detection (AAD) aims to identify the direction of the attended speaker in multi-speaker environments from brain signals, such as Electroencephalography (EEG) signals. However, existing EEG-based AAD methods overlook the spatio-temporal dependencies of EEG signals, limiting their decoding and generalization abilities. To address these issues, this paper proposes a Lightweight Spatio-Temporal Enhancement Nested Network (ListenNet) for AAD. The ListenNet has three key components: Spatio-temporal Dependency Encoder (STDE), Multi-scale Temporal Enhancement (MSTE), and Cross-Nested Attention (CNA). The STDE reconstructs dependencies between consecutive time windows across channels, improving the robustness of dynamic pattern extraction. The MSTE captures temporal features at multiple scales to represent both fine-grained and long-range temporal patterns. In addition, the CNA integrates hierarchical features more effectively through novel dynamic attention mechanisms to capture deep spatio-temporal correlations. Experimental results on three public datasets demonstrate the superiority of ListenNet over state-of-the-art methods in both subject-dependent and challenging subject-independent settings, while reducing the trainable parameter count by approximately 7 times. Code is available at:https://github.com/fchest/ListenNet.

Figures (3)

• Figure 1: Spatio-temporal modeling methods for AAD. Existing methods typically treat space and time separately, processing them from (a) to (b). The proposed ListenNet introduces a multi-scale of temporal patterns, as shown in (c), by considering cross-channel dependencies, temporal dynamics, and spatio-temporal dependencies for a more comprehensive modeling approach.
• Figure 2: The overall structure of our ListenNet for AAD consists of three modules: (a) STDE module, (b) MSTE module, where $k_i$ ($i \in \{1, 2, 3, 4\}$) represents the kernel size used in the dilated convolution, and (c) CNA module, where $E_t'$ and $E_s'$ are depth-aligned input feature maps. The model inputs are normalized and Euclidean-aligned EEG signals, and the outputs are two predicted labels related to auditory attention obtained through a classifier applied to the CNA output features.
• Figure 3: The t-SNE visualization of different types of features on the KUL dataset under the subject-independent condition. Different colors represent different subjects. Circles and squares denote attention to the left or right speaker, respectively.