Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Rotary Steering with Joint Autoregression for Robust Extraction of Closely Moving Speakers in Dynamic Scenarios

Jakob Kienegger, Timo Gerkmann

TL;DR

The paper tackles dynamic target-speaker extraction in Ambisonics, where moving speakers challenge traditional spatial filtering. It generalizes rotary steering to dynamic conditions and introduces weak guidance based on the target's initial direction, complemented by a joint autoregressive framework (AR-SSF and AR-TST) that leverages the enhanced output as temporal-spectral guidance. Across synthetic three-speaker datasets and real-room recordings, the joint autoregressive approach yields substantial improvements in tracking and separation, particularly for closely spaced sources, and outperforms comparable non-autoregressive methods while remaining robust to crossings and distance variations. This work broadens the applicability of rotary steering, offering a practical, architecture-agnostic solution for robust, real-time target extraction in dynamic multi-speaker environments.

Abstract

Latest advances in deep spatial filtering for Ambisonics demonstrate strong performance in stationary multi-speaker scenarios by rotating the sound field toward a target speaker prior to multi-channel enhancement. For applicability in dynamic acoustic conditions with moving speakers, we propose to automate this rotary steering using an interleaved tracking algorithm conditioned on the target's initial direction. However, for nearby or crossing speakers, robust tracking becomes difficult and spatial cues less effective for enhancement. By incorporating the processed recording as additional guide into both algorithms, our novel joint autoregressive framework leverages temporal-spectral correlations of speech to resolve spatially challenging speaker constellations. Consequently, our proposed method significantly improves tracking and enhancement of closely spaced speakers, consistently outperforming comparable non-autoregressive methods on a synthetic dataset. Real-world recordings complement these findings in complex scenarios with multiple speaker crossings and varying speaker-to-array distances.

Adaptive Rotary Steering with Joint Autoregression for Robust Extraction of Closely Moving Speakers in Dynamic Scenarios

TL;DR

The paper tackles dynamic target-speaker extraction in Ambisonics, where moving speakers challenge traditional spatial filtering. It generalizes rotary steering to dynamic conditions and introduces weak guidance based on the target's initial direction, complemented by a joint autoregressive framework (AR-SSF and AR-TST) that leverages the enhanced output as temporal-spectral guidance. Across synthetic three-speaker datasets and real-room recordings, the joint autoregressive approach yields substantial improvements in tracking and separation, particularly for closely spaced sources, and outperforms comparable non-autoregressive methods while remaining robust to crossings and distance variations. This work broadens the applicability of rotary steering, offering a practical, architecture-agnostic solution for robust, real-time target extraction in dynamic multi-speaker environments.

Abstract

Latest advances in deep spatial filtering for Ambisonics demonstrate strong performance in stationary multi-speaker scenarios by rotating the sound field toward a target speaker prior to multi-channel enhancement. For applicability in dynamic acoustic conditions with moving speakers, we propose to automate this rotary steering using an interleaved tracking algorithm conditioned on the target's initial direction. However, for nearby or crossing speakers, robust tracking becomes difficult and spatial cues less effective for enhancement. By incorporating the processed recording as additional guide into both algorithms, our novel joint autoregressive framework leverages temporal-spectral correlations of speech to resolve spatially challenging speaker constellations. Consequently, our proposed method significantly improves tracking and enhancement of closely spaced speakers, consistently outperforming comparable non-autoregressive methods on a synthetic dataset. Real-world recordings complement these findings in complex scenarios with multiple speaker crossings and varying speaker-to-array distances.
Paper Structure (13 sections, 6 equations, 5 figures, 1 table)

This paper contains 13 sections, 6 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Ambisonics signal model for target speaker extraction
  4. Rotary steering for stationary speaker extraction
  5. Proposed Method
  6. Adaptive and fixed rotary steering for dynamic scenarios
  7. Weakly guided adaptive rotary steering framework
  8. Additional guidance via temporal-spectral autoregression
  9. Experimental Setup
  10. Dataset
  11. Algorithm implementation and optimization details
  12. Results
  13. Conclusion

Figures (5)

  • Figure 1: Weakly guided speaker extraction pipelines conditioned on the target's initial doa $(\theta_0, \phi_0)$. Rotary steering is used to direct the spatial filter (SSF) either constantly toward the initial doa (fixed) or use tracking (TST) to adjust the steering to the speaker's movement (adaptive).
  • Figure 2: Synthetic three-speaker (//) dataset modeling continuous motion diaz21srp_phat. Tracking performance is shown for azimuth doa $\theta_t$ using non-AR () and our AR modification () of SELDnet yasuda24causal_seldnet.
  • Figure 3: Tracking (mae) and enhancement (pesq) performance dependency of McNet on the distance to the closest interfering speaker. We report the sample mean with 95% confidence interval error bars.
  • Figure 4: Strongly guided (, ), fixed (, ) and adaptive (, , ) weakly guided extraction using different enhancement algorithms. We report the sample mean with 95% confidence interval error bars.
  • Figure 5: Sample means of unprocessed () and adaptive, weakly guided extraction methods (, , ) evaluated on real-world recordings regarding speech quality (NISQA) and intelligibility (WER).