SlimSpeech: Lightweight and Efficient Text-to-Speech with Slim Rectified Flow

TL;DR

SlimSpeech tackles the challenge of high computational cost in flow-based text-to-speech by marrying rectified flow with a teacher–student distillation framework. A large 1-rectified flow teacher is used to train a compact decoder via annealing reflow and flow-guided distillation, aided by depthwise separable convolutions in the encoder. On LJSpeech, SlimSpeech achieves comparable synthesis quality to larger models with a fraction of parameters, delivering competitive one-step performance and advantageous CPU efficiency, while maintaining GPU speed. The approach demonstrates that high-quality, fast TTS is attainable with substantial model compression and principled distillation, and it remains effective when scaling to additional inference steps for potential performance gains.

Abstract

Recently, flow matching based speech synthesis has significantly enhanced the quality of synthesized speech while reducing the number of inference steps. In this paper, we introduce SlimSpeech, a lightweight and efficient speech synthesis system based on rectified flow. We have built upon the existing speech synthesis method utilizing the rectified flow model, modifying its structure to reduce parameters and serve as a teacher model. By refining the reflow operation, we directly derive a smaller model with a more straight sampling trajectory from the larger model, while utilizing distillation techniques to further enhance the model performance. Experimental results demonstrate that our proposed method, with significantly reduced model parameters, achieves comparable performance to larger models through one-step sampling.

Figures (1)

• Figure 1: The training process of our proposed method.