Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Brain-Informed Speech Separation for Cochlear Implants

Tom Gajecki, Jonas Althoff, Waldo Nogueira

TL;DR

This work tackles the cocktail-party problem in cochlear implants by introducing a brain-informed speech separation framework that leverages EEG-derived attention cues to steer processing toward the attended speaker, yielding an attended electrodogram for CI stimulation with a fixed 2 ms latency. It compares an audio-only baseline to a brain-informed model that fuses audio and EEG features in a lightweight fusion layer, producing a single attended output to resolve label-permutation ambiguity. A mixed curriculum training strategy enhances robustness to degraded attention cues, enabling stable gains across varying cue quality. Evaluations on Libri2Mix show consistent SIR improvements and improved electrode-wise temporal fidelity, highlighting the practicality of integrating neural cues into cognitively adaptive CI processing for real-world listening conditions.

Abstract

We propose a brain-informed speech separation method for cochlear implants (CIs) that uses electroencephalography (EEG)-derived attention cues to guide enhancement toward the attended speaker. An attention-guided network fuses audio mixtures with EEG features through a lightweight fusion layer, producing attended-source electrodograms for CI stimulation while resolving the label-permutation ambiguity of audio-only separators. Robustness to degraded attention cues is improved with a mixed curriculum that varies cue quality during training, yielding stable gains even when EEG-speech correlation is moderate. In multi-talker conditions, the model achieves higher signal-to-interference ratio improvements than an audio-only electrodogram baseline while remaining slightly smaller (167k vs. 171k parameters). With 2 ms algorithmic latency and comparable cost, the approach highlights the promise of coupling auditory and neural cues for cognitively adaptive CI processing.

Brain-Informed Speech Separation for Cochlear Implants

TL;DR

This work tackles the cocktail-party problem in cochlear implants by introducing a brain-informed speech separation framework that leverages EEG-derived attention cues to steer processing toward the attended speaker, yielding an attended electrodogram for CI stimulation with a fixed 2 ms latency. It compares an audio-only baseline to a brain-informed model that fuses audio and EEG features in a lightweight fusion layer, producing a single attended output to resolve label-permutation ambiguity. A mixed curriculum training strategy enhances robustness to degraded attention cues, enabling stable gains across varying cue quality. Evaluations on Libri2Mix show consistent SIR improvements and improved electrode-wise temporal fidelity, highlighting the practicality of integrating neural cues into cognitively adaptive CI processing for real-world listening conditions.

Abstract

We propose a brain-informed speech separation method for cochlear implants (CIs) that uses electroencephalography (EEG)-derived attention cues to guide enhancement toward the attended speaker. An attention-guided network fuses audio mixtures with EEG features through a lightweight fusion layer, producing attended-source electrodograms for CI stimulation while resolving the label-permutation ambiguity of audio-only separators. Robustness to degraded attention cues is improved with a mixed curriculum that varies cue quality during training, yielding stable gains even when EEG-speech correlation is moderate. In multi-talker conditions, the model achieves higher signal-to-interference ratio improvements than an audio-only electrodogram baseline while remaining slightly smaller (167k vs. 171k parameters). With 2 ms algorithmic latency and comparable cost, the approach highlights the promise of coupling auditory and neural cues for cognitively adaptive CI processing.
Paper Structure (16 sections, 13 equations, 4 figures)

This paper contains 16 sections, 13 equations, 4 figures.

Table of Contents

  1. INTRODUCTION
  2. METHODS & MATERIALS
  3. Notation
  4. System Architecture
  5. Baseline Model (audio-only, two outputs)
  6. Brain-Informed Model (single attended output)
  7. Curriculum Learning Strategy
  8. EEG Envelope Processing (proxy attention cue)
  9. Loss and Training
  10. Evaluation Metrics
  11. Dataset
  12. RESULTS
  13. Overall Performance Comparison
  14. Cue Correlation Analysis
  15. Linear Cross-Correlation Analysis
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Overview of the baseline speech separation system (left) and brain-informed speech enhancement system (right). The brain-informed system combines audio mixture signals with an EEG-derived attention cue to generate attended-source electrodograms for CI stimulation.
  • Figure 2: SIR improvement of baseline vs. brain-informed model across input SIR levels. We assume oracle proxy attention cues for the brain-informed model.
  • Figure 3: SIR improvement as a function of cue correlation for different curriculum strategies. Mixed curriculum shows consistent performance across correlation levels. Error bars indicate standard error.
  • Figure 4: LCCs per electrode for different curriculum strategies and the baseline. Higher LCCs indicate stronger preservation of attended-speech temporal patterns, relevant for intelligibility in CI stimulation.