Leveraging Spatial Cues from Cochlear Implant Microphones to Efficiently Enhance Speech Separation in Real-World Listening Scenes

TL;DR

The paper addresses the challenge of front-end speech separation for cochlear implant users in real-world, reverberant environments. It evaluates how spatial information intrinsic to CI microphone signals (implicit cues) and additional explicit spatial features (IPD) can improve separation when using an efficient time-domain model, SuDoRM-RF, adapted for multi-channel CI inputs. The authors generate a realistic spatialized two-talker dataset using RIRs, CI-HRTFs, and BRIRs (59,688 BRIRs) and show that real-world acoustics significantly degrade non-spatial models, but spatial cues markedly boost performance, especially for same-gender talkers or weak spectral differences. The findings provide guidance on when to rely on implicit cues versus when to add explicit IPD information, highlighting implications for low-latency, energy-efficient front-end processing in CIs and other assistive devices. Overall, the work demonstrates that leveraging naturally occurring spatial cues in CI recordings supports robust speech separation in everyday listening scenes while maintaining computational efficiency.

Abstract

Speech separation approaches for single-channel, dry speech mixtures have significantly improved. However, real-world spatial and reverberant acoustic environments remain challenging, limiting the effectiveness of these approaches for assistive hearing devices like cochlear implants (CIs). To address this, we quantify the impact of real-world acoustic scenes on speech separation and explore how spatial cues can enhance separation quality efficiently. We analyze performance based on implicit spatial cues (inherent in the acoustic input and learned by the model) and explicit spatial cues (manually calculated spatial features added as auxiliary inputs). Our findings show that spatial cues (both implicit and explicit) improve separation for mixtures with spatially separated and nearby talkers. Furthermore, spatial cues enhance separation when spectral cues are ambiguous, such as when voices are similar. Explicit spatial cues are particularly beneficial when implicit spatial cues are weak. For instance, single CI microphone recordings provide weaker implicit spatial cues than bilateral CIs, but even single CIs benefit from explicit cues. These results emphasize the importance of training models on real-world data to improve generalizability in everyday listening scenarios. Additionally, our statistical analyses offer insights into how data properties influence model performance, supporting the development of efficient speech separation approaches for CIs and other assistive devices in real-world settings.

Figures (5)

• Figure 1: (A) Schematic depiction of data generation. CI-HRIR = Cochlear implants Head Impulse Response; RIR = Room impulse response; BRIR = Binaural Room Impulse Response. (B) An acoustic scene depicting talkers positioned around a listener located in a room. (C) Schematic example of an acoustic scene with two talkers and a listener wearing bilateral CIs. Each CI has three microphones: F = front microphone, B = back microphone and T = T-microphone.
• Figure 2: (A) Reverberation time ($T_{60}$) as a function of room size. Each circle represents a single room. (B) Effect of reverberation and spatialization on speech mixtures. Top panes show waveforms of a dry, non-spatial speech mixture and a spatial, reverberant version of the same speech mixture. For illustration, bottom panes depict spectrograms (but note that models are trained directly on the waveform). (C) Presence of implicit spatial cues in real-world acoustic scenes. Top pane shows an example of a two-talker mixture in a spatial, reverberant scene. One talker is at -90$^{\circ}$, the other talker is at +90$^{\circ}$. Bottom pane visualizes the corresponding interaural level differences for this speech mixture.
• Figure 3: (A) SuDoRM-RF tzinis2022compute implementation in present study. Top row depicts SuDoRM-RF approach for input configurations consisting only of the speech mixture (i.e. waveform). Bottom row depicts our SuDoRM-RF approach for input configurations consisting of speech mixture (i.e. waveform) and IPD as an auxiliary feature. (B) Schematic overview of all input configurations. Color indicates the presence of implicit (blue) and combined implicit and explicit (green) spatial cues. A higher saturation signifies stronger spatial cues.
• Figure 4: Speech separation performance as a function of spatial separation angle between talkers. Each panel depicts the results for one input configuration (Fig. 3B, Table \ref{['tab:train-test main result']}). Colors indicate implicit (blue) and combined implicit and explicit (green) spatial information load, with higher saturation indicating higher spatial information load (similar to Fig. 3B). Bars represent average SI-SDR, error bars reflect standard error of the mean. As a reference, the gray dashed line indicates the average SI-SDR at a separation angle of 0°, that is, no spatial distance. Asterisks reflect a statistically significant difference in SI-SDR at that relevant separation angle and the 0° reference: * = p$<$ 0.05, ** = p$<$ 0.01.
• Figure 5: SI-SDR as a function of talker gender pairing and spatial distance for various input configurations. Bars represent average SI-SDR at small spatial distances (0° and 15°, filled bars) or and large spatial distances (60° and 90°, open bars) across the different gender pairings (red = two female talkers; blue = two male talkers; purple = one male and one female talker). Asterisks reflect a statistically significant difference (Kruskal-Wallis H tests, FDR corrected): *** = p$<$ 0.001; ** = p$<$ 0.01.