Instance-Specific Test-Time Training for Speech Editing in the Wild

TL;DR

This work addresses robust speech editing under real-world, variable acoustics by introducing instance-specific test-time training (TTT). The model combines a FluentSpeech-based backbone with a Diffusion Transformer denoiser and a two-stage TTT pipeline that independently tailors the duration predictor and spectrogram denoiser for each test utterance, guided by direct supervision in unedited regions and auxiliary losses in edited regions, including a controllable duration term $\mathcal{L}_{\mathrm{dur}}$. Empirical results on LibriTTS clean and GigaSpeech in-the-wild show state-of-the-art or competitive performance across WER, SIM, MCD, and MOS, with substantial gains from TTT over ablations; ablations demonstrate the importance of the DP/SD components and the chosen losses for intelligibility and naturalness. The approach enables explicit speech-rate control without extra duration modules and improves continuity across edited boundaries, offering practical utility for robust, controllable speech editing in real-world scenarios, while acknowledging latency/compute limitations and the need for safeguards against misuse.

Abstract

Speech editing systems aim to naturally modify speech content while preserving acoustic consistency and speaker identity. However, previous studies often struggle to adapt to unseen and diverse acoustic conditions, resulting in degraded editing performance in real-world scenarios. To address this, we propose an instance-specific test-time training method for speech editing in the wild. Our approach employs direct supervision from ground-truth acoustic features in unedited regions and indirect supervision in edited regions via auxiliary losses based on duration constraints and phoneme prediction. This strategy mitigates the bandwidth discontinuity problem in speech editing, ensuring smooth acoustic transitions between unedited and edited regions. Additionally, it enables precise control over speech rate by adapting the model to target durations via mask length adjustment during test-time training. Experiments on in-the-wild benchmark datasets demonstrate that our method outperforms existing speech editing systems in both objective and subjective evaluations.

Figures (4)

• Figure 1: Overview of the proposed framework. In subfigure (a), LR denotes the length regulator. In subfigure (b), $M$ indicates the length of the edit mask, which is required for test-time training (TTT) of the duration predictor. In subfigures (b) and (c), "Masking for TTT" refers to randomly masking unedited regions to compute the reconstruction loss during TTT. Red boxes indicate edit regions, and yellow boxes represent randomly masked regions for TTT. The flame icon denotes modules that are updated during TTT, whereas the snowflake icon indicates modules whose parameters remain frozen.
• Figure 2: Mel-spectrograms of generated speech at different speech rates using TTT for the duration predictor. The middle row represents the original sentence duration, while the top and bottom rows show $-$20% and $+$20% adjustments, respectively. Red boxes indicate the edited regions.
• Figure 3: Linear spectrograms of ground-truth and generated speech from different systems. The top panel shows the full spectrogram, while the bottom panel highlights the corresponding regions with red boxes.
• Figure 4: Instruction interfaces for subjective evaluation tasks.