Beyond Speaker Identity: Text Guided Target Speech Extraction
Mingyue Huo, Abhinav Jain, Cong Phuoc Huynh, Fanjie Kong, Pichao Wang, Zhu Liu, Vimal Bhat
TL;DR
This work tackles target speech extraction without relying on enrollment or visual cues by introducing a text-guided approach. The StyleTSE framework couples a SepFormer-based separation network with a bi-modality clue network that fuses text and audio clues through a gated mechanism, enabling extraction conditioned on speaking style as described in natural language. The TextrolMix dataset provides a large, diverse benchmark with natural language style descriptions and style-sharing reference audio, enabling robust training and evaluation. Key findings show that text guidance can match or exceed traditional cues in guiding separation, with strong performance when text describes speaking style and substantial gains when combined with audio clues, demonstrating the practical potential of text-based controls for TSE in real-world scenarios where enrollment cues are unavailable.
Abstract
Target Speech Extraction (TSE) traditionally relies on explicit clues about the speaker's identity like enrollment audio, face images, or videos, which may not always be available. In this paper, we propose a text-guided TSE model StyleTSE that uses natural language descriptions of speaking style in addition to the audio clue to extract the desired speech from a given mixture. Our model integrates a speech separation network adapted from SepFormer with a bi-modality clue network that flexibly processes both audio and text clues. To train and evaluate our model, we introduce a new dataset TextrolMix with speech mixtures and natural language descriptions. Experimental results demonstrate that our method effectively separates speech based not only on who is speaking, but also on how they are speaking, enhancing TSE in scenarios where traditional audio clues are absent. Demos are at: https://mingyue66.github.io/TextrolMix/demo/