CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models

TL;DR

CosyEdit presents an end-to-end speech editing model that eliminates external alignment modules by post-training a CosyVoice-based AR+NAR system on a 250-hour GigaEdit dataset. It jointly fine-tunes a large language model for autoregressive speech token generation and a guided flow-matching model (OT-CFM) with reference paths to preserve timbre and acoustic detail. Through zero-shot in-context training and one-shot inference, CosyEdit internalizes speech-text alignment and achieves strong RealEdit performance, outperforming several baselines and rivaling state-of-the-art cascades. The work demonstrates a practical, scalable approach to robust speech editing and outlines future directions in safety, multilinguality, and finer-grained control.

Abstract

Automatic speech editing aims to modify spoken content based on textual instructions, yet traditional cascade systems suffer from complex preprocessing pipelines and a reliance on explicit external temporal alignment. Addressing these limitations, we propose CosyEdit, an end-to-end speech editing model adapted from CosyVoice through task-specific fine-tuning and an optimized inference procedure, which internalizes speech-text alignment while ensuring high consistency between the speech before and after editing. By fine-tuning on only 250 hours of supervised data from our curated GigaEdit dataset, our 400M-parameter model achieves reliable speech editing performance. Experiments on the RealEdit benchmark indicate that CosyEdit not only outperforms several billion-parameter language model baselines but also matches the performance of state-of-the-art cascade approaches. These results demonstrate that, with task-specific fine-tuning and inference optimization, robust and efficient speech editing capabilities can be unlocked from a zero-shot TTS model, yielding a novel and cost-effective end-to-end solution for high-quality speech editing.

Figures (3)

• Figure 1: Comparison between cascade and end-to-end speech editing. Italicized characters indicate speech segments not temporally aligned with the text, while upright characters denote segments with established alignment timestamps. Red blank rectangular boxes represent masked speech tokens to be edited.
• Figure 2: (a) is an example of four editing tasks for constructing the speech editing training dataset GigaEdit. (b) is a schematic diagram of CosyEdit. , and represent the markers of "start of the sequence", "end of the sequence" and "turn of speech" respectively. The dotted line represents the autoregressive decoding in the reasoning stage. (c) provides an enlarged view of our flow matching model conditioning on a speaker embedding $\mathbf{v}$, semantic tokens $\mu_Z$ represents the concatenation of $\mu_X$ and $\mu_Y$, $\tilde{Z}$ represents the concatenation of speech features $X$ and full masked speech features $\tilde{Y}$, and intermediate state $Z_t$ at timestep $t$ on the probabilistic density path.
• Figure 3: (a) is the input format during training. (b) is the input format for speech editing inference.