Rethinking Discrete Speech Representation Tokens for Accent Generation

TL;DR

This work addresses the underexplored encoding of accent information in discrete speech representation tokens (DSRTs) by introducing Accent ABX as a measure of accessibility and cross-accent VC resynthesis as a gauge of recoverability. Through systematic analysis of DSRTs derived from HuBERT, HuBERT-ft, and Whisper discretised with RepCodec, the study reveals that accent cues are most prominent in mid-early layers and are significantly diminished by ASR supervision, while naive codebook size reductions fail to disentangle accent from content and speaker. Based on these findings, the authors propose content-only and content-accent DSRTs, which demonstrate superior controllable accent generation in VC compared to existing designs, and emphasize the need for accent-aware evaluation in DSRT design. The work provides practical guidance for building DSRTs that support accent-controlled speech generation and highlights avenues for extending the framework to zero-shot TTS and multilingual settings, with anticipated broad impact on inclusive speech technology and potential safeguards against misuse.

Abstract

Discrete Speech Representation Tokens (DSRTs) have become a foundational component in speech generation. While prior work has extensively studied phonetic and speaker information in DSRTs, how accent information is encoded in DSRTs remains largely unexplored. In this paper, we present the first systematic investigation of accent information in DSRTs. We propose a unified evaluation framework that measures both accessibility of accent information via a novel Accent ABX task and recoverability via cross-accent Voice Conversion (VC) resynthesis. Using this framework, we analyse DSRTs derived from a variety of speech encoders. Our results reveal that accent information is substantially reduced when ASR supervision is used to fine-tune the encoder, but cannot be effectively disentangled from phonetic and speaker information through naive codebook size reduction. Based on these findings, we propose new content-only and content-accent DSRTs that significantly outperform existing designs in controllable accent generation. Our work highlights the importance of accent-aware evaluation and provides practical guidance for designing DSRTs for accent-controlled speech generation.

Figures (5)

• Figure 1: Proposed pipeline for evaluating the recoverability and accessibility of accent, speaker, and phonetic information in various Discrete Speech Representation Tokens (DSRTs).
• Figure 2: Accent, speaker, and phonetic information in different DSRTs across layers and representations. All codebook sizes and code dimensions are set to 1024 for fair comparison. (See Figure \ref{['fig:results_layers_full']} in Appendix \ref{['sec:full_results']} for additional results from cross-accent VC.)
• Figure 3: Accent, speaker, and phonetic information recoverability in different DSRTs across codebook sizes. (See Figure \ref{['fig:results_codebooksizes_full']} in Appendix \ref{['sec:full_results']} for additional results from cross-accent VC and ABX.)
• Figure 4: Accent, speaker, and phonetic information in different DSRTs across layers and representations. Full version of Figure \ref{['fig:main_plot']}.
• Figure 5: Accent, speaker, and phonetic information in different DSRTs across codebook sizes. Full version of Figure \ref{['fig:recoverability_with_cbz']}.