Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech

Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, Jingchen Shu

TL;DR

IndexTTS2 tackles the challenge of precise duration control in autoregressive TTS while enhancing emotional expressiveness in zero-shot settings. It introduces a cascaded architecture (T2S, S2M, Vocoder) with a duration-aware token system, GRL-based emotion disentanglement, a Text-to-Emotion module, and GPT latent enhancement, trained via a three-stage scheme. The model achieves state-of-the-art performance on multiple benchmarks, demonstrates near-perfect duration accuracy, and delivers superior emotional fidelity and speaker similarity, aided by NL emotion control and efficient latent fusion. These advances enable tightly synchronized, emotionally rich voice synthesis for applications like video dubbing and expressive narration, with practical training and inference strategies for robustness.

Abstract

Existing autoregressive large-scale text-to-speech (TTS) models have advantages in speech naturalness, but their token-by-token generation mechanism makes it difficult to precisely control the duration of synthesized speech. This becomes a significant limitation in applications requiring strict audio-visual synchronization, such as video dubbing. This paper introduces IndexTTS2, which proposes a novel, general, and autoregressive model-friendly method for speech duration control. The method supports two generation modes: one explicitly specifies the number of generated tokens to precisely control speech duration; the other freely generates speech in an autoregressive manner without specifying the number of tokens, while faithfully reproducing the prosodic features of the input prompt. Furthermore, IndexTTS2 achieves disentanglement between emotional expression and speaker identity, enabling independent control over timbre and emotion. In the zero-shot setting, the model can accurately reconstruct the target timbre (from the timbre prompt) while perfectly reproducing the specified emotional tone (from the style prompt). To enhance speech clarity in highly emotional expressions, we incorporate GPT latent representations and design a novel three-stage training paradigm to improve the stability of the generated speech. Additionally, to lower the barrier for emotional control, we designed a soft instruction mechanism based on text descriptions by fine-tuning Qwen3, effectively guiding the generation of speech with the desired emotional orientation. Finally, experimental results on multiple datasets show that IndexTTS2 outperforms state-of-the-art zero-shot TTS models in terms of word error rate, speaker similarity, and emotional fidelity. Audio samples are available at: https://index-tts.github.io/index-tts2.github.io/

IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech

TL;DR

IndexTTS2 tackles the challenge of precise duration control in autoregressive TTS while enhancing emotional expressiveness in zero-shot settings. It introduces a cascaded architecture (T2S, S2M, Vocoder) with a duration-aware token system, GRL-based emotion disentanglement, a Text-to-Emotion module, and GPT latent enhancement, trained via a three-stage scheme. The model achieves state-of-the-art performance on multiple benchmarks, demonstrates near-perfect duration accuracy, and delivers superior emotional fidelity and speaker similarity, aided by NL emotion control and efficient latent fusion. These advances enable tightly synchronized, emotionally rich voice synthesis for applications like video dubbing and expressive narration, with practical training and inference strategies for robustness.

Abstract

Existing autoregressive large-scale text-to-speech (TTS) models have advantages in speech naturalness, but their token-by-token generation mechanism makes it difficult to precisely control the duration of synthesized speech. This becomes a significant limitation in applications requiring strict audio-visual synchronization, such as video dubbing. This paper introduces IndexTTS2, which proposes a novel, general, and autoregressive model-friendly method for speech duration control. The method supports two generation modes: one explicitly specifies the number of generated tokens to precisely control speech duration; the other freely generates speech in an autoregressive manner without specifying the number of tokens, while faithfully reproducing the prosodic features of the input prompt. Furthermore, IndexTTS2 achieves disentanglement between emotional expression and speaker identity, enabling independent control over timbre and emotion. In the zero-shot setting, the model can accurately reconstruct the target timbre (from the timbre prompt) while perfectly reproducing the specified emotional tone (from the style prompt). To enhance speech clarity in highly emotional expressions, we incorporate GPT latent representations and design a novel three-stage training paradigm to improve the stability of the generated speech. Additionally, to lower the barrier for emotional control, we designed a soft instruction mechanism based on text descriptions by fine-tuning Qwen3, effectively guiding the generation of speech with the desired emotional orientation. Finally, experimental results on multiple datasets show that IndexTTS2 outperforms state-of-the-art zero-shot TTS models in terms of word error rate, speaker similarity, and emotional fidelity. Audio samples are available at: https://index-tts.github.io/index-tts2.github.io/

Paper Structure

This paper contains 25 sections, 5 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Precise Duration Control for Large-Scale TTS.
  4. Emotionally Controllable Large-Scale TTS.
  5. Proposed Method
  6. Autoregressive Text-to-Semantic Module (T2S)
  7. Duration Control:
  8. Emotional Control:
  9. Training and Inference:
  10. Semantic-to-Mel Module (S2M)
  11. GPT Latent Enhancement:
  12. Training and Inference:
  13. Text-to-Emotion (T2E)
  14. Experiments
  15. Experimental Settings
  16. ...and 10 more sections

Figures (4)

  • Figure 1: The overview of IndexTTS2.
  • Figure 2: Autoregressive Text-to-Semantic Module. When speech token num is specified, precise control of the number of synthesized semantic tokens is performed. The emotion adapter (red dashed lines) is employed to extract emotional features from the style prompt, which are then used as input to the Text-to-Semantic process for the reconstruction of emotions.
  • Figure 3: Semantic-to-Mel module based on flow matching.
  • Figure 4: Comparison of WER for duration control section.