Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis

TL;DR

Cross-Lingual F5-TTS eliminates the need for audio prompt transcripts in flow-matching TTS by leveraging MMS forced alignment to obtain word boundaries for transcript-free training. It introduces speaking-rate predictors at phoneme, syllable, and word levels to derive target durations without transcripts, using a Gaussian Cross-Entropy loss to capture ordinal rate information. Empirically, the approach achieves comparable performance to F5-TTS on standard English tasks while enabling cross-lingual voice cloning to unseen languages, with finer granularity predictors offering the best duration accuracy. This work expands cross-lingual TTS capabilities and demonstrates robust generalization, though it acknowledges limitations in transferring nuanced speaker traits like accents and emotions without transcripts.

Abstract

Flow-matching-based text-to-speech (TTS) models have shown high-quality speech synthesis. However, most current flow-matching-based TTS models still rely on reference transcripts corresponding to the audio prompt for synthesis. This dependency prevents cross-lingual voice cloning when audio prompt transcripts are unavailable, particularly for unseen languages. The key challenges for flow-matching-based TTS models to remove audio prompt transcripts are identifying word boundaries during training and determining appropriate duration during inference. In this paper, we introduce Cross-Lingual F5-TTS, a framework that enables cross-lingual voice cloning without audio prompt transcripts. Our method preprocesses audio prompts by forced alignment to obtain word boundaries, enabling direct synthesis from audio prompts while excluding transcripts during training. To address the duration modeling challenge, we train speaking rate predictors at different linguistic granularities to derive duration from speaker pace. Experiments show that our approach matches the performance of F5-TTS while enabling cross-lingual voice cloning.

Figures (2)

• Figure 1: Training framework of Cross-Lingual F5-TTS. MMS forced alignment produces word boundaries for training data, where the left segment serves as transcript-free audio prompt and the mel spectrogram of the right segment is masked for prediction.
• Figure 2: Training pipeline of the speaking rate predictor. The model predicts discrete rate categories from mel-spectrograms, while ground truth speaking rates are mapped to nearest categories for GCE loss computation.