Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TFGA-Net: Temporal-Frequency Graph Attention Network for Brain-Controlled Speaker Extraction

Youhao Si, Yuan Liao, Qiushi Han, Yuhang Yang, Rui Dai, Liya Huang

TL;DR

The paper addresses EEG-guided target speech extraction in multi-talker environments by directly modeling the listener's attentional focus from EEG. It introduces TFGA-Net, which combines an EEG encoder capable of multi-scale time--frequency representation and cortical-topology processing with a MossFormer2-based separator to capture global context and speech rhythms, followed by a speech decoder. The method achieves state-of-the-art performance on Cocktail Party and KUL datasets, notably improving SI-SDR while also delivering competitive perceptual metrics. Ablation studies demonstrate the value of the temporal-frequency graph attention EEG encoding and the MossFormer2 module for robust, real-time target speech extraction guided by neural signals. This work advances brain-controlled hearing applications by providing a robust, EEG-driven architecture for selective listening in complex acoustic scenes.

Abstract

The rapid development of auditory attention decoding (AAD) based on electroencephalography (EEG) signals offers the possibility EEG-driven target speaker extraction. However, how to effectively utilize the target-speaker common information between EEG and speech remains an unresolved problem. In this paper, we propose a model for brain-controlled speaker extraction, which utilizes the EEG recorded from the listener to extract the target speech. In order to effectively extract information from EEG signals, we derive multi-scale time--frequency features and further incorporate cortical topological structures that are selectively engaged during the task. Moreover, to effectively exploit the non-Euclidean structure of EEG signals and capture their global features, the graph convolutional networks and self-attention mechanism are used in the EEG encoder. In addition, to make full use of the fused EEG and speech feature and preserve global context and capture speech rhythm and prosody, we introduce MossFormer2 which combines MossFormer and RNN-Free Recurrent as separator. Experimental results on both the public Cocktail Party and KUL dataset in this paper show that our TFGA-Net model significantly outper-forms the state-of-the-art method in certain objective evaluation metrics. The source code is available at: https://github.com/LaoDa-X/TFGA-NET.

TFGA-Net: Temporal-Frequency Graph Attention Network for Brain-Controlled Speaker Extraction

TL;DR

The paper addresses EEG-guided target speech extraction in multi-talker environments by directly modeling the listener's attentional focus from EEG. It introduces TFGA-Net, which combines an EEG encoder capable of multi-scale time--frequency representation and cortical-topology processing with a MossFormer2-based separator to capture global context and speech rhythms, followed by a speech decoder. The method achieves state-of-the-art performance on Cocktail Party and KUL datasets, notably improving SI-SDR while also delivering competitive perceptual metrics. Ablation studies demonstrate the value of the temporal-frequency graph attention EEG encoding and the MossFormer2 module for robust, real-time target speech extraction guided by neural signals. This work advances brain-controlled hearing applications by providing a robust, EEG-driven architecture for selective listening in complex acoustic scenes.

Abstract

The rapid development of auditory attention decoding (AAD) based on electroencephalography (EEG) signals offers the possibility EEG-driven target speaker extraction. However, how to effectively utilize the target-speaker common information between EEG and speech remains an unresolved problem. In this paper, we propose a model for brain-controlled speaker extraction, which utilizes the EEG recorded from the listener to extract the target speech. In order to effectively extract information from EEG signals, we derive multi-scale time--frequency features and further incorporate cortical topological structures that are selectively engaged during the task. Moreover, to effectively exploit the non-Euclidean structure of EEG signals and capture their global features, the graph convolutional networks and self-attention mechanism are used in the EEG encoder. In addition, to make full use of the fused EEG and speech feature and preserve global context and capture speech rhythm and prosody, we introduce MossFormer2 which combines MossFormer and RNN-Free Recurrent as separator. Experimental results on both the public Cocktail Party and KUL dataset in this paper show that our TFGA-Net model significantly outper-forms the state-of-the-art method in certain objective evaluation metrics. The source code is available at: https://github.com/LaoDa-X/TFGA-NET.
Paper Structure (14 sections, 12 equations, 3 figures, 2 tables)

This paper contains 14 sections, 12 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. METHODS
  3. Problem Formulation
  4. Overall Architecture
  5. Loss Function
  6. EXPERIMENTS
  7. Datasets
  8. Data Processing
  9. Implementation Details
  10. Evaluation Metrics
  11. RESULTS
  12. Comparative Analysis
  13. Ablation Study
  14. CONCLUSION

Figures (3)

  • Figure 1: The overall block diagram of the proposed TFGA-Net model.
  • Figure 2: The overall block diagram of the proposed EEGEncode.
  • Figure 3: Performance on Speaker Extraction Network