Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis
Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, Yijin Xing
TL;DR
Fish-Speech presents a multilingual, non-G2P TTS framework that combines a serial fast-slow Dual-AR architecture with a GFSQ-based Firefly-GAN vocoder and LLM-driven linguistic feature extraction. The Dual-AR design stacks a Slow Transformer and a Fast Transformer to stabilize codebook generation and improve efficiency, while FF-GAN achieves high-fidelity synthesis with near-100% codebook utilization. Trained on ~720k hours of multilingual data, the system demonstrates superior performance in handling complex linguistic features, voice cloning, and cross-language synthesis, with strong objective and subjective metrics. The work foregrounds real-time, scalable TTS suitable for AI agents and multilingual applications, and provides open-source access for broader adoption and further research.
Abstract
Text-to-Speech (TTS) systems face ongoing challenges in processing complex linguistic features, handling polyphonic expressions, and producing natural-sounding multilingual speech - capabilities that are crucial for future AI applications. In this paper, we present Fish-Speech, a novel framework that implements a serial fast-slow Dual Autoregressive (Dual-AR) architecture to enhance the stability of Grouped Finite Scalar Vector Quantization (GFSQ) in sequence generation tasks. This architecture improves codebook processing efficiency while maintaining high-fidelity outputs, making it particularly effective for AI interactions and voice cloning. Fish-Speech leverages Large Language Models (LLMs) for linguistic feature extraction, eliminating the need for traditional grapheme-to-phoneme (G2P) conversion and thereby streamlining the synthesis pipeline and enhancing multilingual support. Additionally, we developed FF-GAN through GFSQ to achieve superior compression ratios and near 100\% codebook utilization. Our approach addresses key limitations of current TTS systems while providing a foundation for more sophisticated, context-aware speech synthesis. Experimental results show that Fish-Speech significantly outperforms baseline models in handling complex linguistic scenarios and voice cloning tasks, demonstrating its potential to advance TTS technology in AI applications. The implementation is open source at \href{https://github.com/fishaudio/fish-speech}{https://github.com/fishaudio/fish-speech}.