Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TokenChain: A Discrete Speech Chain via Semantic Token Modeling

Mingxuan Wang, Satoshi Nakamura

TL;DR

TokenChain presents a fully discrete speech chain that binds a semantic-token ASR with a two-stage TTS (autoregressive T2S and masked S2A) in a closed loop, enabling end-to-end feedback via ST estimators and dynamic loss balancing. The approach leverages semantic token representations to jointly train recognition and token-conditioned generation, achieving faster convergence (2–6 epochs) and substantial WER reductions on LibriSpeech and TED-LIUM with limited forgetting. Quantitative results show 5–13% lower equal-epoch error and up to 56% ASR and 31% T2S WER reductions under domain adaptation, demonstrating robust cross-domain transfer. The work highlights the viability and benefits of token-centric, discrete interfaces for bidirectional speech chain learning and efficient optimization in modern ASR/TTS pipelines.

Abstract

Machine Speech Chain, simulating the human perception-production loop, proves effective in jointly improving ASR and TTS. We propose TokenChain, a fully discrete speech chain coupling semantic-token ASR with a two-stage TTS: an autoregressive text-to-semantic model co-trained with ASR and a masked-generative semantic-to-acoustic model for synthesis only. End-to-end feedback across the text interface is enabled with straight-through argmax/Gumbel-Softmax and balanced with supervised ASR via dynamic weight averaging. Ablations examine optimal temperature schedules for in- and cross-domain transfer. Evaluation reveals TokenChain surpasses baseline accuracy 2-6 epochs earlier and yields 5-13% lower equal-epoch error with stable T2S on LibriSpeech, and reduces relative ASR WER by 56% and T2S WER by 31% on TED-LIUM with minimal forgetting, showing that chain learning remains effective with token interfaces and models.

TokenChain: A Discrete Speech Chain via Semantic Token Modeling

TL;DR

TokenChain presents a fully discrete speech chain that binds a semantic-token ASR with a two-stage TTS (autoregressive T2S and masked S2A) in a closed loop, enabling end-to-end feedback via ST estimators and dynamic loss balancing. The approach leverages semantic token representations to jointly train recognition and token-conditioned generation, achieving faster convergence (2–6 epochs) and substantial WER reductions on LibriSpeech and TED-LIUM with limited forgetting. Quantitative results show 5–13% lower equal-epoch error and up to 56% ASR and 31% T2S WER reductions under domain adaptation, demonstrating robust cross-domain transfer. The work highlights the viability and benefits of token-centric, discrete interfaces for bidirectional speech chain learning and efficient optimization in modern ASR/TTS pipelines.

Abstract

Machine Speech Chain, simulating the human perception-production loop, proves effective in jointly improving ASR and TTS. We propose TokenChain, a fully discrete speech chain coupling semantic-token ASR with a two-stage TTS: an autoregressive text-to-semantic model co-trained with ASR and a masked-generative semantic-to-acoustic model for synthesis only. End-to-end feedback across the text interface is enabled with straight-through argmax/Gumbel-Softmax and balanced with supervised ASR via dynamic weight averaging. Ablations examine optimal temperature schedules for in- and cross-domain transfer. Evaluation reveals TokenChain surpasses baseline accuracy 2-6 epochs earlier and yields 5-13% lower equal-epoch error with stable T2S on LibriSpeech, and reduces relative ASR WER by 56% and T2S WER by 31% on TED-LIUM with minimal forgetting, showing that chain learning remains effective with token interfaces and models.

Paper Structure

This paper contains 15 sections, 11 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Data Preparation
  4. Discrete ASR (Semantic Tokens→Text)
  5. Autoregressive T2S (Text→Semantic Tokens)
  6. Discrete Pass-Through and Chain Feedback
  7. Final Objective with DWA Scheduling
  8. NAR S2A (Semantic→Acoustic Tokens)
  9. Experiments
  10. Setup
  11. Model and Training Details
  12. Main Results on LibriSpeech
  13. Domain Adaptation on TED-LIUM
  14. Conclusions
  15. Acknowledgments

Figures (3)

  • Figure 1: Illustration of proposed TokenChain framework. AR = autoregressive, NAR = non-autoregressive. The codec emits a $T\times Q$ token-index matrix ($Q$ = RVQ stages/codebooks).
  • Figure 2: Plot of LibriSpeech learning curves: CER (left), WER (right). TokenChain (red) vs. baseline (blue): surpasses the baseline 2–6 epochs earlier and achieves lower final error.
  • Figure 3: Illustration of Domain Behavior: relative change in correct rate (higher is better). Top: major TED-LIUM gains; bottom: minor LibriSpeech performance degradations.