Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models

Xinting Liao, Ruinan Jin, Hanlin Yu, Deval Pandya, Xiaoxiao Li

TL;DR

RVCBench addresses the gap in evaluating robustness of modern voice cloning systems under deployment-like shifts across the full generation pipeline. It proposes a four-dimensional evaluation framework, curates a dataset of 225 speakers and 14,370 utterances across 11 VC models, and defines 10 tasks to probe input, generation, output, and perturbation robustness. The study uncovers substantial robustness gaps: input shifts degrade content fidelity; long-form and cross-lingual generation amplifies instability; post-processing and perturbations undermine both perceptual quality and detectability. The open-source benchmark provides a standardized testbed to drive the development of more robust and trustworthy VC models.

Abstract

Modern voice cloning (VC) can synthesize speech that closely matches a target speaker from only seconds of reference audio, enabling applications such as personalized speech interfaces and dubbing. In practical deployments, modern audio generation models inevitably encounter noisy reference audios, imperfect text prompts, and diverse downstream processing, which can significantly hurt robustness. Despite rapid progress in VC driven by autoregressive codec-token language models and diffusion-based models, robustness under realistic deployment shifts remains underexplored. This paper introduces RVCBench, a comprehensive benchmark that evaluates Robustness in VC across the full generation pipeline, including input variation, generation challenges, output post-processing, and adversarial perturbations, covering 10 robustness tasks, 225 speakers, 14,370 utterances, and 11 representative modern VC models. Our evaluation uncovers substantial robustness gaps in VC: performance can deteriorate sharply under common input shifts and post-processing; long-context and cross-lingual scenarios further expose stability limitations; and both passive noise and proactive perturbation influence generation robustness. Collectively, these findings provide a unified picture of how current VC models fail in practice and introduce a standardized, open-source testbed to support the development of more robust and deployable VC models. We open-source our project at https://github.com/Nanboy-Ronan/RVCBench.

RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models

TL;DR

RVCBench addresses the gap in evaluating robustness of modern voice cloning systems under deployment-like shifts across the full generation pipeline. It proposes a four-dimensional evaluation framework, curates a dataset of 225 speakers and 14,370 utterances across 11 VC models, and defines 10 tasks to probe input, generation, output, and perturbation robustness. The study uncovers substantial robustness gaps: input shifts degrade content fidelity; long-form and cross-lingual generation amplifies instability; post-processing and perturbations undermine both perceptual quality and detectability. The open-source benchmark provides a standardized testbed to drive the development of more robust and trustworthy VC models.

Abstract

Modern voice cloning (VC) can synthesize speech that closely matches a target speaker from only seconds of reference audio, enabling applications such as personalized speech interfaces and dubbing. In practical deployments, modern audio generation models inevitably encounter noisy reference audios, imperfect text prompts, and diverse downstream processing, which can significantly hurt robustness. Despite rapid progress in VC driven by autoregressive codec-token language models and diffusion-based models, robustness under realistic deployment shifts remains underexplored. This paper introduces RVCBench, a comprehensive benchmark that evaluates Robustness in VC across the full generation pipeline, including input variation, generation challenges, output post-processing, and adversarial perturbations, covering 10 robustness tasks, 225 speakers, 14,370 utterances, and 11 representative modern VC models. Our evaluation uncovers substantial robustness gaps in VC: performance can deteriorate sharply under common input shifts and post-processing; long-context and cross-lingual scenarios further expose stability limitations; and both passive noise and proactive perturbation influence generation robustness. Collectively, these findings provide a unified picture of how current VC models fail in practice and introduce a standardized, open-source testbed to support the development of more robust and deployable VC models. We open-source our project at https://github.com/Nanboy-Ronan/RVCBench.
Paper Structure (59 sections, 7 equations, 12 figures, 28 tables)

This paper contains 59 sections, 7 equations, 12 figures, 28 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Benchmaring VC and audio robustness.
  4. RVCBench
  5. Benchmark Overview
  6. Input Robustness
  7. Generation Robustness
  8. Output Robustness
  9. Audio Perturbation Robustness
  10. Experiments Analysis
  11. Overall performance of Voice Cloning
  12. Degradation under Input Shifts
  13. Instability in Diversified Generation Tasks
  14. Sensitivity over Perceptual in Outputs
  15. Vulnerability in Perturbation and Counteraction
  16. ...and 44 more sections

Figures (12)

  • Figure 1: Overview of RVCBench. We benchmark robustness on VC across the full generation pipeline along four robustness dimensions, i.e., inputs, generation, outputs, and audio perturbations.
  • Figure 2: Performance on different accents.
  • Figure 3: Performance change on text-prompt shifts.
  • Figure 4: Performance over prompt audio length.
  • Figure 5: Performance on long-text generation.
  • ...and 7 more figures