Advancing Topic Segmentation of Broadcasted Speech with Multilingual Semantic Embeddings

TL;DR

This work tackles broadcast news topic segmentation by evaluating both a traditional pipeline (ASR + text segmentation) and a direct end-to-end approach that operates on semantic representations derived from multilingual speech encoders. It introduces a large, multilingual dataset derived from Euronews (six European languages plus Hindi) and analyzes cross-domain and zero-shot performance using pretrained SONAR embeddings and Conformer-based architectures. Key findings show that while the pipeline remains strong for English, the end-to-end approach is competitive and benefits from multilingual training, with notable cross-lingual and cross-domain capabilities. The authors provide model and data preparation resources to foster open research in multilingual spoken news topic segmentation, highlighting practical implications for scalable, multilingual multimedia processing.

Abstract

Recent advancements in speech-based topic segmentation have highlighted the potential of pretrained speech encoders to capture semantic representations directly from speech. Traditionally, topic segmentation has relied on a pipeline approach in which transcripts of the automatic speech recognition systems are generated, followed by text-based segmentation algorithms. In this paper, we introduce an end-to-end scheme that bypasses this conventional two-step process by directly employing semantic speech encoders for segmentation. Focused on the broadcasted news domain, which poses unique challenges due to the diversity of speakers and topics within single recordings, we address the challenge of accessing topic change points efficiently in an end-to-end manner. Furthermore, we propose a new benchmark for spoken news topic segmentation by utilizing a dataset featuring approximately 1000 hours of publicly available recordings across six European languages and including an evaluation set in Hindi to test the model's cross-domain performance in a cross-lingual, zero-shot scenario. This setup reflects real-world diversity and the need for models adapting to various linguistic settings. Our results demonstrate that while the traditional pipeline approach achieves a state-of-the-art $P_k$ score of 0.2431 for English, our end-to-end model delivers a competitive $P_k$ score of 0.2564. When trained multilingually, these scores further improve to 0.1988 and 0.2370, respectively. To support further research, we release our model along with data preparation scripts, facilitating open research on multilingual spoken news topic segmentation.

Figures (3)

• Figure 1: Our pipeline system.
• Figure 2: Our end-to-end system.
• Figure 3: The (a) is an acceptable boundary, (b) represents false negative where the true segment is present but is not predicted, and (c) represents a false alarm where the true segment doesnot exist but is predicted.