Lightweight End-to-end Text-to-speech Synthesis for low resource on-device applications
Biel Tura Vecino, Adam Gabryś, Daniel Mątwicki, Andrzej Pomirski, Tom Iddon, Marius Cotescu, Jaime Lorenzo-Trueba
TL;DR
LE2E introduces a compact end-to-end TTS architecture that jointly trains a lightweight acoustic latent model with a neural vocoder, guided by GAN-based discriminators for on-device synthesis. It achieves MOS 3.79 on LJSpeech while using only 3.71M parameters and a real-time factor of 0.0084, placing it near state-of-the-art in quality but far more efficient. The study shows that end-to-end training on a lightweight architecture can outperform a two-stage cascade on the same backbone and demonstrates practical viability for real-time, offline TTS on low-resource devices.
Abstract
Recent works have shown that modelling raw waveform directly from text in an end-to-end (E2E) fashion produces more natural-sounding speech than traditional neural text-to-speech (TTS) systems based on a cascade or two-stage approach. However, current E2E state-of-the-art models are computationally complex and memory-consuming, making them unsuitable for real-time offline on-device applications in low-resource scenarios. To address this issue, we propose a Lightweight E2E-TTS (LE2E) model that generates high-quality speech requiring minimal computational resources. We evaluate the proposed model on the LJSpeech dataset and show that it achieves state-of-the-art performance while being up to $90\%$ smaller in terms of model parameters and $10\times$ faster in real-time-factor. Furthermore, we demonstrate that the proposed E2E training paradigm achieves better quality compared to an equivalent architecture trained in a two-stage approach. Our results suggest that LE2E is a promising approach for developing real-time, high quality, low-resource TTS applications for on-device applications.