Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FAC-FACodec: Controllable Zero-Shot Foreign Accent Conversion with Factorized Speech Codec

Yurii Halychanskyi, Cameron Churchwell, Yutong Wen, Volodymyr Kindratenko

TL;DR

FAC-FACodec introduces an explicit, user-controllable degree of accent modification for foreign accent conversion by learning a native-pronunciation prior over FACodec content latents through diffusion and applying inference-time noise and denoising. The framework operates with non-parallel data, using native speech with transcripts for training and non-native speech with transcripts for inference, and concentrates on pronunciation while preserving suprasegmental cues. Objective and subjective evaluations on L2-Arctic demonstrate competitive performance and a smooth continuum of accent strength guided by the control knob $t_{\text{start}}$, highlighting practical applicability in learning, dubbing, and personal communication. The approach offers a unique capability among prior AC methods by enabling tunable trade-offs between accent conversion and speaker identity, with future work targeting perceptually guided noise and coordinated prosody control.

Abstract

Previous accent conversion (AC) methods, including foreign accent conversion (FAC), lack explicit control over the degree of modification. Because accent modification can alter the perceived speaker identity, balancing conversion strength and identity preservation is crucial. We present an AC framework that provides an explicit, user-controllable parameter for accent modification. The method targets pronunciation while preserving suprasegmental cues such as intonation and phoneme durations. Results show performance comparable to recent AC systems, stronger preservation of speaker identity, and unique support for controllable accent conversion.

FAC-FACodec: Controllable Zero-Shot Foreign Accent Conversion with Factorized Speech Codec

TL;DR

FAC-FACodec introduces an explicit, user-controllable degree of accent modification for foreign accent conversion by learning a native-pronunciation prior over FACodec content latents through diffusion and applying inference-time noise and denoising. The framework operates with non-parallel data, using native speech with transcripts for training and non-native speech with transcripts for inference, and concentrates on pronunciation while preserving suprasegmental cues. Objective and subjective evaluations on L2-Arctic demonstrate competitive performance and a smooth continuum of accent strength guided by the control knob , highlighting practical applicability in learning, dubbing, and personal communication. The approach offers a unique capability among prior AC methods by enabling tunable trade-offs between accent conversion and speaker identity, with future work targeting perceptually guided noise and coordinated prosody control.

Abstract

Previous accent conversion (AC) methods, including foreign accent conversion (FAC), lack explicit control over the degree of modification. Because accent modification can alter the perceived speaker identity, balancing conversion strength and identity preservation is crucial. We present an AC framework that provides an explicit, user-controllable parameter for accent modification. The method targets pronunciation while preserving suprasegmental cues such as intonation and phoneme durations. Results show performance comparable to recent AC systems, stronger preservation of speaker identity, and unique support for controllable accent conversion.

Paper Structure

This paper contains 20 sections, 10 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Noise-Controlled Translation with Diffusion Priors
  4. FACodec: Factorized Neural Speech Codec
  5. Proposed Framework
  6. Forward Diffusion Process
  7. Conditional Diffusion Model
  8. Posterior Derivation
  9. Inference Procedure and Accent-Strength Control
  10. Second Residual Prediction and Training Objective
  11. Experiments
  12. Dataset
  13. Model Architecture and Training Configuration
  14. Baseline Models
  15. Evaluation Metrics
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Illustration of how $t_{\text{start}}$ controls a single-frame posterior. Larger $t_{\text{start}}$ widens the likelihood and shifts the posterior toward the native prior mean, while smaller $t_{\text{start}}$ keeps it concentrated near the non-native input.
  • Figure 2: MUSHRA-like listening test results. Violin plots show subjective ratings of accent similarity to Standard American English for reconstruction and for denoising with $t_{\text{start}}=25,50,75,100$. Higher scores indicate stronger perceived similarity.