FreeV: Free Lunch For Vocoders Through Pseudo Inversed Mel Filter

TL;DR

This paper tackles the efficiency bottleneck of frequency-domain vocoders by introducing FreeV, which uses a pseudo-inverse Mel-filter-based amplitude prior to initialize the amplitude spectrum $\hat{A}$ via $\hat{A}=\max(\operatorname{Abs}(M^+X),10^{-5})$. By freezing $M^+$ and modeling only the residual between $\hat{A}$ and the true amplitude, FreeV substantially reduces the ASP parameter count and accelerates inference, achieving about $1.8\times$ faster GPU speed and roughly half the parameters of APNet2, while improving reconstruction quality on LJSpeech. The method also shows faster convergence and robust gains when using the estimated amplitude as input for other vocoders. Overall, FreeV advances real-time, high-fidelity speech synthesis by integrating principled signal-processing priors into a streamlined spectral vocoder framework.

Abstract

Vocoders reconstruct speech waveforms from acoustic features and play a pivotal role in modern TTS systems. Frequent-domain GAN vocoders like Vocos and APNet2 have recently seen rapid advancements, outperforming time-domain models in inference speed while achieving comparable audio quality. However, these frequency-domain vocoders suffer from large parameter sizes, thus introducing extra memory burden. Inspired by PriorGrad and SpecGrad, we employ pseudo-inverse to estimate the amplitude spectrum as the initialization roughly. This simple initialization significantly mitigates the parameter demand for vocoder. Based on APNet2 and our streamlined Amplitude prediction branch, we propose our FreeV, compared with its counterpart APNet2, our FreeV achieves 1.8 times inference speed improvement with nearly half parameters. Meanwhile, our FreeV outperforms APNet2 in resynthesis quality, marking a step forward in pursuing real-time, high-fidelity speech synthesis. Code and checkpoints is available at: https://github.com/BakerBunker/FreeV

Figures (5)

• Figure 1: Inference speed and reconstruction performance of multiple methods trained and evaluated on LJSpeech. The size of the circle represents the model parameter size. FreeV achieves the fastest inference speed and reconstruction quality with half parameter size compared to APNet2.
• Figure 2: The overall architecture of FreeV, the amplitude prediction branch (ASP) of APNet2, which has red background, is replaced by a more lightweight architecture with green background.
• Figure 3: Comparison of real log amplitude spectra $A$ and estimated log spectra $\hat{A}$.
• Figure 4: Loss curves of APNet2 (orange) and FreeV (blue).
• Figure 5: Early stage mel loss curves of multiple models trained with (blue) and without estimated amplitude spectra $\hat{A}$ (orange).