Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS

Haibin Wu, Bach Viet Do, Naveen Suda, Julian Chan, Madhavan C R, Gene-Ping Yang, Yi-Chiao Wu, Naoyuki Kanda, Yossef Adi, Xin Lei, Yue Liu, Florian Metze, Yuzong Liu

TL;DR

The paper tackles the latency bottleneck of Mimi-based on-device TTS by substituting its deconvolution upsampling with a purely transformer-based decoder, inspired by TS3-Codec. It demonstrates that the T-Mimi archi- tecture can reduce on-device decoding latency from 42.1 ms to 4.4 ms for 80 ms audio frames, enabling real-time streaming on mobile hardware. Through quantization-aware training, it reveals that final waveform-proximate layers must remain in full precision to preserve audio quality, guiding a selective-precision strategy that reduces storage significantly while maintaining performance. The results show competitive or superior perceptual quality with substantial efficiency gains, offering a practical, generalizable path for deploying convolution-based neural audio codecs on resource-constrained devices.

Abstract

Neural audio codecs provide promising acoustic features for speech synthesis, with representative streaming codecs like Mimi providing high-quality acoustic features for real-time Text-to-Speech (TTS) applications. However, Mimi's decoder, which employs a hybrid transformer and convolution architecture, introduces significant latency bottlenecks on edge devices due to the the compute intensive nature of deconvolution layers which are not friendly for mobile-CPUs, such as the most representative framework XNNPACK. This paper introduces T-Mimi, a novel modification of the Mimi codec decoder that replaces its convolutional components with a purely transformer-based decoder, inspired by the TS3-Codec architecture. This change dramatically reduces on-device TTS latency from 42.1ms to just 4.4ms. Furthermore, we conduct quantization aware training and derive a crucial finding: the final two transformer layers and the concluding linear layers of the decoder, which are close to the waveform, are highly sensitive to quantization and must be preserved at full precision to maintain audio quality.

T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS

TL;DR

The paper tackles the latency bottleneck of Mimi-based on-device TTS by substituting its deconvolution upsampling with a purely transformer-based decoder, inspired by TS3-Codec. It demonstrates that the T-Mimi archi- tecture can reduce on-device decoding latency from 42.1 ms to 4.4 ms for 80 ms audio frames, enabling real-time streaming on mobile hardware. Through quantization-aware training, it reveals that final waveform-proximate layers must remain in full precision to preserve audio quality, guiding a selective-precision strategy that reduces storage significantly while maintaining performance. The results show competitive or superior perceptual quality with substantial efficiency gains, offering a practical, generalizable path for deploying convolution-based neural audio codecs on resource-constrained devices.

Abstract

Neural audio codecs provide promising acoustic features for speech synthesis, with representative streaming codecs like Mimi providing high-quality acoustic features for real-time Text-to-Speech (TTS) applications. However, Mimi's decoder, which employs a hybrid transformer and convolution architecture, introduces significant latency bottlenecks on edge devices due to the the compute intensive nature of deconvolution layers which are not friendly for mobile-CPUs, such as the most representative framework XNNPACK. This paper introduces T-Mimi, a novel modification of the Mimi codec decoder that replaces its convolutional components with a purely transformer-based decoder, inspired by the TS3-Codec architecture. This change dramatically reduces on-device TTS latency from 42.1ms to just 4.4ms. Furthermore, we conduct quantization aware training and derive a crucial finding: the final two transformer layers and the concluding linear layers of the decoder, which are close to the waveform, are highly sensitive to quantization and must be preserved at full precision to maintain audio quality.
Paper Structure (19 sections, 1 figure, 4 tables)

This paper contains 19 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Codec feature as a good medium for TTS
  4. On-phone TTS requirements
  5. Quantization aware training
  6. Method
  7. Architecture
  8. Training
  9. Quatization aware training setup
  10. Experiments
  11. Experimental setup
  12. Training setup
  13. Evaluation setup
  14. Experimental results
  15. CMOS between non-quantized Mimi and T-Mimi
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Model architectures for the original Mimi decoder (upper) and the T-Mimi decoder (bottom).