Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Edit Content, Preserve Acoustics: Imperceptible Text-Based Speech Editing via Self-Consistency Rewards

Yong Ren, Jiangyan Yi, Jianhua Tao, Zhengqi Wen, Tao Wang

TL;DR

This paper tackles imperceptible text-based speech editing by decoupling linguistic content changes from acoustic rendering through a two-stage framework: Structural Foundations, which performs semantic-space editing with a Prefix-Suffix-Middle scheme, and Flow Matching for acoustic reconstruction; and Perceptual Alignment, which uses Self-Consistency Rewards GRPO with a pre-trained TTS critic and ASR/WER plus duration constraints to ensure seamless fusion with context. The methodへ utilizes a semantic-token-based LLM for infilling, a Flow Matching decoder for waveform synthesis, and a GRPO-based reinforcement learning objective that integrates a log-probability reward from the TTS model and an intelligibility reward from ASR, gated by validity checks. Experiments on LibriHeavy and Ming-based benchmarks show significant improvements over state-of-the-art autoregressive and non-autoregressive baselines in WER, speaker similarity, and perceptual quality, especially for longer edits. Overall, the work demonstrates that editing in a disentangled semantic space, paired with perceptual alignment, yields robust, intelligible, and natural-sounding edits suitable for real-world post-production contexts.

Abstract

Imperceptible text-based speech editing allows users to modify spoken content by altering the transcript. It demands that modified segments fuse seamlessly with the surrounding context. Prevalent methods operating in the acoustic space suffer from inherent content-style entanglement, leading to generation instability and boundary artifacts. In this paper, we propose a novel framework grounded in the principle of "Edit Content, Preserve Acoustics". Our approach relies on two core components: (1) Structural Foundations, which decouples editing into a stable semantic space while delegating acoustic reconstruction to a Flow Matching decoder; and (2) Perceptual Alignment, which employs a novel Self-Consistency Rewards Group Relative Policy Optimization. By leveraging a pre-trained Text-to-Speech model as an implicit critic -- complemented by strict intelligibility and duration constraints -- we effectively align the edited semantic token sequence with the original context. Empirical evaluations demonstrate that our method significantly outperforms state-of-the-art autoregressive and non-autoregressive baselines, achieving superior intelligibility, robustness, and perceptual quality.

Edit Content, Preserve Acoustics: Imperceptible Text-Based Speech Editing via Self-Consistency Rewards

TL;DR

This paper tackles imperceptible text-based speech editing by decoupling linguistic content changes from acoustic rendering through a two-stage framework: Structural Foundations, which performs semantic-space editing with a Prefix-Suffix-Middle scheme, and Flow Matching for acoustic reconstruction; and Perceptual Alignment, which uses Self-Consistency Rewards GRPO with a pre-trained TTS critic and ASR/WER plus duration constraints to ensure seamless fusion with context. The methodへ utilizes a semantic-token-based LLM for infilling, a Flow Matching decoder for waveform synthesis, and a GRPO-based reinforcement learning objective that integrates a log-probability reward from the TTS model and an intelligibility reward from ASR, gated by validity checks. Experiments on LibriHeavy and Ming-based benchmarks show significant improvements over state-of-the-art autoregressive and non-autoregressive baselines in WER, speaker similarity, and perceptual quality, especially for longer edits. Overall, the work demonstrates that editing in a disentangled semantic space, paired with perceptual alignment, yields robust, intelligible, and natural-sounding edits suitable for real-world post-production contexts.

Abstract

Imperceptible text-based speech editing allows users to modify spoken content by altering the transcript. It demands that modified segments fuse seamlessly with the surrounding context. Prevalent methods operating in the acoustic space suffer from inherent content-style entanglement, leading to generation instability and boundary artifacts. In this paper, we propose a novel framework grounded in the principle of "Edit Content, Preserve Acoustics". Our approach relies on two core components: (1) Structural Foundations, which decouples editing into a stable semantic space while delegating acoustic reconstruction to a Flow Matching decoder; and (2) Perceptual Alignment, which employs a novel Self-Consistency Rewards Group Relative Policy Optimization. By leveraging a pre-trained Text-to-Speech model as an implicit critic -- complemented by strict intelligibility and duration constraints -- we effectively align the edited semantic token sequence with the original context. Empirical evaluations demonstrate that our method significantly outperforms state-of-the-art autoregressive and non-autoregressive baselines, achieving superior intelligibility, robustness, and perceptual quality.
Paper Structure (11 sections, 8 equations, 1 figure, 2 tables)

This paper contains 11 sections, 8 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overall architecture
  4. Structural Foundations: Semantic Space Editing
  5. Perceptual Alignment: Self-Consistency Rewards GRPO
  6. Experiments
  7. Experimental Settings
  8. Implementation Details
  9. Main Results Analysis
  10. Robustness on Edited Duration
  11. Conclusion

Figures (1)

  • Figure 1: The overall framework of our proposed method. The pipeline consists of two stages: (1) Structural Foundations (Left), where we employ a semantic-token-based LLM for conditional content infilling followed by a Flow Matching decoder for acoustic reconstruction; and (2) Perceptual Alignment (Right), where the policy is fine-tuned via Self-Consistency Rewards GRPO.