Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extraction

TL;DR

Contextual Speech Extraction (CSE) addresses extracting a target speech from a mixture using only textual dialogue history as an implicit cue. The authors propose three CSE architectures—cascaded, unified (ContSep, ContExt), and hybrid (H-ContExt)—based on a Sepformer backbone and validate on three datasets (DailyTalk, SpokenWOZ, TED-LIUM 3). They demonstrate that with only two turns of history, ACC can exceed $>90\%$ and that combining textual context with enrollment cues yields further gains. The findings suggest a practical, flexible approach for real-world applications such as mobile messaging where explicit cues are costly to acquire.

Abstract

In this paper, we investigate a novel approach for Target Speech Extraction (TSE), which relies solely on textual context to extract the target speech. We refer to this task as Contextual Speech Extraction (CSE). Unlike traditional TSE methods that rely on pre-recorded enrollment utterances, video of the target speaker's face, spatial information, or other explicit cues to identify the target stream, our proposed method requires only a few turns of previous dialogue (or monologue) history. This approach is naturally feasible in mobile messaging environments where voice recordings are typically preceded by textual dialogue that can be leveraged implicitly. We present three CSE models and analyze their performances on three datasets. Through our experiments, we demonstrate that even when the model relies purely on dialogue history, it can achieve over 90 % accuracy in identifying the correct target stream with only two previous dialogue turns. Furthermore, we show that by leveraging both textual context and enrollment utterances as cues during training, we further enhance our model's flexibility and effectiveness, allowing us to use either cue during inference, or combine both for improved performance. Samples and code available on https://miraodasilva.github.io/cse-project-page .

Figures (4)

• Figure 1: Overall concept of the proposed contextual speech extraction framework: By employing dialogue history as a cue, the model can extract the target speech from a mixture without explicit cues.
• Figure 2: Illustration of the cascaded CSE pipeline.
• Figure 3: Illustration of the proposed CSE methods. (a) ContSep that separates all streams from the mixture and predicts the target stream, (b) ContExt and H-ContExt that produce the target stream only. For simplicity, the encoder and decoder are omitted in the figure.
• Figure 4: Performances (Left: SI-SNR i, Right: ACC) of Unified CSE and Hybrid CSE on different context lengths on SpokenWOZ dataset. "Full" indicates that we use the full context, without limiting its length.