Advanced Zero-Shot Text-to-Speech for Background Removal and Preservation with Controllable Masked Speech Prediction

TL;DR

Zero-shot TTS systems struggle when prompts contain environmental backgrounds, requiring controllable background removal or preservation. The authors propose Controllable Masked Speech Prediction (CMSP) together with a Dual Speaker Encoder, guided by binary control signals $c$, to jointly model background removal and preservation. The CMSP objective combines Background Removal and Background Preservation losses to steer dual-masked predictions, while the dual encoders reduce interference and improve timbre-background fidelity. Experiments on LibriTTS with noise, reverberation, and interfering speech show robust gains in speaker similarity and background fidelity under both in-domain and out-of-domain conditions, enabling precise, tunable background handling in challenging acoustic environments.

Abstract

The acoustic background plays a crucial role in natural conversation. It provides context and helps listeners understand the environment, but a strong background makes it difficult for listeners to understand spoken words. The appropriate handling of these backgrounds is situation-dependent: Although it may be necessary to remove background to ensure speech clarity, preserving the background is sometimes crucial to maintaining the contextual integrity of the speech. Despite recent advancements in zero-shot Text-to-Speech technologies, current systems often struggle with speech prompts containing backgrounds. To address these challenges, we propose a Controllable Masked Speech Prediction strategy coupled with a dual-speaker encoder, utilizing a task-related control signal to guide the prediction of dual background removal and preservation targets. Experimental results demonstrate that our approach enables precise control over the removal or preservation of background across various acoustic conditions and exhibits strong generalization capabilities in unseen scenarios.

Figures (3)

• Figure 1: Model Architecture
• Figure 2: Mean Speaker Similarity Comparison by SNRs for Each Model
• Figure 3: Mel-spectrogram Comparison for Generated Speech with Preserved Background Noise among Models