Speaker Retrieval in the Wild: Challenges, Effectiveness and Robustness

TL;DR

This work tackles speaker retrieval in large, real-world archives where task-specific labels are scarce and acoustic conditions are highly unconstrained. It presents a practical framework that combines pre-processing, PyAnnote-based diarisation, and embedding extraction using $x$-vector, ECAPA-TDNN, and TitaNet, with duration-informed segment weighting and retrieval via cosine similarity, including a concrete representation: $ ext{score}[ ext{fileID}] = \max_{i=1}^{M_{ ext{fileID}}} \text{Sim}(\mathbf{q}, \mathbf{s}^{\text{fileID}}_i) $. The study benchmarks multiple embeddings, analyzes label noise from synopses, and demonstrates robustness across distortions such as additive noise, sampling rate changes, bit-depth reductions, and reverberation, with fusion of ECAPA-TDNN-SB and TitaNet-L offering consistent gains. It highlights practical implications for retrieving target speakers in archives like BBC Rewind and suggests future work in semi-supervised learning and multimodal retrieval to better handle label ambiguities and cross-modal cues.

Abstract

There is a growing abundance of publicly available or company-owned audio/video archives, highlighting the increasing importance of efficient access to desired content and information retrieval from these archives. This paper investigates the challenges, solutions, effectiveness, and robustness of speaker retrieval systems developed "in the wild" which involves addressing two primary challenges: extraction of task-relevant labels from limited metadata for system development and evaluation, as well as the unconstrained acoustic conditions encountered in the archive, ranging from quiet studios to adverse noisy environments. While we focus on the publicly-available BBC Rewind archive (spanning 1948 to 1979), our framework addresses the broader issue of speaker retrieval on extensive and possibly aged archives with no control over the content and acoustic conditions. Typically, these archives offer a brief and general file description, mostly inadequate for specific applications like speaker retrieval, and manual annotation of such large-scale archives is unfeasible. We explore various aspects of system development (e.g., speaker diarisation, embedding extraction, query selection) and analyse the challenges, possible solutions, and their functionality. To evaluate the performance, we conduct systematic experiments in both clean setup and against various distortions simulating real-world applications. Our findings demonstrate the effectiveness and robustness of the developed speaker retrieval systems, establishing the versatility and scalability of the proposed framework for a wide range of applications beyond the BBC Rewind corpus.

Figures (10)

• Figure 1: Workflow of the speaker retrieval system consisting of pre-processing, speaker diarisation, speaker embedding extraction and retrieval stages.
• Figure 2: Extraction of segment ($\mathbf{e}$) and speaker ($\mathbf{s}$) level embeddings using pre-trained diarisation and embedding extraction models from input audio. $N$ and $M$, respectively, denote the number of segments and speakers, both automatically determined by the speaker diarisation block.
• Figure 3: Histogram of the information extracted using diarisation. (a) number of segments (#segments), (b) number of speakers (#speakers), (c) speech ratio. The horizontal lines at 10, 100, and 1000 in (a) and (b) indicate levels to facilitate readability and comparison.
• Figure 4: Histograms of the #speakers per file (according to diarisation) along with the histogram of the #names (according to synopses) per file.
• Figure 5: Occurrences of names vs. name index, computed by calculating the frequency of individual names extracted from Rewind synopses by NER module. In total, NER detected over 5,800 distinct person names.