Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guided Speaker Embedding

Shota Horiguchi, Takafumi Moriya, Atsushi Ando, Takanori Ashihara, Hiroshi Sato, Naohiro Tawara, Marc Delcroix

TL;DR

The paper tackles extracting a target speaker’s embedding from overlapped long-form speech, a key challenge for speaker verification and diarization. It introduces a guided speaker embedding method that augments input features with target and non-target activity cues and masks attention to focus on active target intervals, enabling embedding extraction with arbitrary numbers of interference speakers. The approach builds on ECAPA-TDNN with 80-dim log-Mel inputs and yields 192-dim embeddings trained with additive angular margin loss, validated on VoxCeleb1/2 with three-speaker mixtures. Results show meaningful improvements in both verification (especially in one-vs-many, highly overlapped cases) and diarization (lower DER/JER across diverse datasets), demonstrating practical benefits for real-world multi-speaker systems. The method remains off-the-shelf and avoids auxiliary tasks, making it compatible with existing multi-speaker processing pipelines and opening avenues for joint use with multi-speaker ASR.

Abstract

This paper proposes a guided speaker embedding extraction system, which extracts speaker embeddings of the target speaker using speech activities of target and interference speakers as clues. Several methods for long-form overlapped multi-speaker audio processing are typically two-staged: i) segment-level processing and ii) inter-segment speaker matching. Speaker embeddings are often used for the latter purpose. Typical speaker embedding extraction approaches only use single-speaker intervals to avoid corrupting the embeddings with speech from interference speakers. However, this often makes speaker embeddings impossible to extract because sufficiently long non-overlapping intervals are not always available. In this paper, we propose using speaker activities as clues to extract the embedding of the speaker-of-interest directly from overlapping speech. Specifically, we concatenate the activity of target and non-target speakers to acoustic features before being fed to the model. We also condition the attention weights used for pooling so that the attention weights of the intervals in which the target speaker is inactive are zero. The effectiveness of the proposed method is demonstrated in speaker verification and speaker diarization.

Guided Speaker Embedding

TL;DR

The paper tackles extracting a target speaker’s embedding from overlapped long-form speech, a key challenge for speaker verification and diarization. It introduces a guided speaker embedding method that augments input features with target and non-target activity cues and masks attention to focus on active target intervals, enabling embedding extraction with arbitrary numbers of interference speakers. The approach builds on ECAPA-TDNN with 80-dim log-Mel inputs and yields 192-dim embeddings trained with additive angular margin loss, validated on VoxCeleb1/2 with three-speaker mixtures. Results show meaningful improvements in both verification (especially in one-vs-many, highly overlapped cases) and diarization (lower DER/JER across diverse datasets), demonstrating practical benefits for real-world multi-speaker systems. The method remains off-the-shelf and avoids auxiliary tasks, making it compatible with existing multi-speaker processing pipelines and opening avenues for joint use with multi-speaker ASR.

Abstract

This paper proposes a guided speaker embedding extraction system, which extracts speaker embeddings of the target speaker using speech activities of target and interference speakers as clues. Several methods for long-form overlapped multi-speaker audio processing are typically two-staged: i) segment-level processing and ii) inter-segment speaker matching. Speaker embeddings are often used for the latter purpose. Typical speaker embedding extraction approaches only use single-speaker intervals to avoid corrupting the embeddings with speech from interference speakers. However, this often makes speaker embeddings impossible to extract because sufficiently long non-overlapping intervals are not always available. In this paper, we propose using speaker activities as clues to extract the embedding of the speaker-of-interest directly from overlapping speech. Specifically, we concatenate the activity of target and non-target speakers to acoustic features before being fed to the model. We also condition the attention weights used for pooling so that the attention weights of the intervals in which the target speaker is inactive are zero. The effectiveness of the proposed method is demonstrated in speaker verification and speaker diarization.

Paper Structure

This paper contains 10 sections, 7 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Conventional method: Basic speaker embedding
  5. Proposed method: Guided speaker embedding
  6. Experimental setup
  7. Results
  8. Speaker verification
  9. Speaker diarization
  10. Conclusion

Figures (3)

  • Figure 1: Schematic diagram of guided speaker embedding (Target: Speaker 2).
  • Figure 2: Statistics of target speaker's utterances ($n=37\,611$).
  • Figure 3: Examples of attention weights calculated using the proposed method. Red and gray bars visualize the speech activities of target and interference speakers, respectively.