MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement

TL;DR

MF-Speech tackles the dual challenges of entangled speech factors and coarse control by introducing a factor-purifying MF-SpeechEncoder and a fine-grained MF-SpeechGenerator. The encoder produces independent content, timbre, and emotion representations, while the generator uses dynamic fusion and Hierarchical Style Adaptive Normalization to enable precise, compositional control during waveform synthesis. Across reconstruction and multi-factor compositional tasks, MF-Speech achieves strong objective and subjective results, including low WER and high style similarity, and demonstrates robust factor transferability. This framework advances controllable speech synthesis with disentangled factors and transferable representations suitable for diverse speakers and emotions.

Abstract

Generating expressive and controllable human speech is one of the core goals of generative artificial intelligence, but its progress has long been constrained by two fundamental challenges: the deep entanglement of speech factors and the coarse granularity of existing control mechanisms. To overcome these challenges, we have proposed a novel framework called MF-Speech, which consists of two core components: MF-SpeechEncoder and MF-SpeechGenerator. MF-SpeechEncoder acts as a factor purifier, adopting a multi-objective optimization strategy to decompose the original speech signal into highly pure and independent representations of content, timbre, and emotion. Subsequently, MF-SpeechGenerator functions as a conductor, achieving precise, composable and fine-grained control over these factors through dynamic fusion and Hierarchical Style Adaptive Normalization (HSAN). Experiments demonstrate that in the highly challenging multi-factor compositional speech generation task, MF-Speech significantly outperforms current state-of-the-art methods, achieving a lower word error rate (WER=4.67%), superior style control (SECS=0.5685, Corr=0.68), and the highest subjective evaluation scores(nMOS=3.96, sMOS_emotion=3.86, sMOS_style=3.78). Furthermore, the learned discrete factors exhibit strong transferability, demonstrating their significant potential as a general-purpose speech representation.

Figures (5)

• Figure 1: MF-Speech enables independent and fine-grained control over speech content, timbre, and emotion factors for speech synthesis.
• Figure 2: The training process of MF-Speech consists of three stages. The first stage ensures high-precision conversion between waveforms and features. The second stage disentangles clean and independent content, timbre, and emotion factors. The third stage enables fine-grained, multi-factor control for waveform generation.
• Figure 3: The related architectures and data flows of dynamic fusion and HSAN
• Figure 4: t-SNE visualization of timbre and emotion representation from four models.
• Figure 5: The ablation results of SpeechGenerator.