Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding

Rui-Chen Zheng, Wenrui Liu, Hui-Peng Du, Qinglin Zhang, Chong Deng, Qian Chen, Wen Wang, Yang Ai, Zhen-Hua Ling

TL;DR

VARSTok tackles the mismatch between speech information density and fixed-token-rate tokenizers by introducing a fully dynamic, variable-frame-rate tokenizer that uses a temporal-aware density peak clustering algorithm and an implicit duration coding scheme to embed both content and duration into a single token ID. A single-codebook VQ framework, expanded token vocabulary, and an end-to-end training objective enable efficient, duration-aware tokens that can be directly used by autoregressive speech language models without extra duration predictors. Across LibriTTS reconstruction, ARCH semantic evaluation, and zero-shot TTS tasks, VARSTok achieves competitive or superior performance at lower average token rates (around $30.95$ Hz) compared with fixed-rate baselines, while improving perceptual naturalness and maintaining low WER. This work demonstrates that fully dynamic, duration-aware acoustic tokenization can be seamlessly integrated into downstream speech LMs, offering token efficiency and improved prosody without architectural changes.”

Abstract

Existing speech tokenizers typically assign a fixed number of tokens per second, regardless of the varying information density or temporal fluctuations in the speech signal. This uniform token allocation mismatches the intrinsic structure of speech, where information is distributed unevenly over time. To address this, we propose VARSTok, a VAriable-frame-Rate Speech Tokenizer that adapts token allocation based on local feature similarity. VARSTok introduces two key innovations: (1) a temporal-aware density peak clustering algorithm that adaptively segments speech into variable-length units, and (2) a novel implicit duration coding scheme that embeds both content and temporal span into a single token index, eliminating the need for auxiliary duration predictors. Extensive experiments show that VARSTok significantly outperforms strong fixed-rate baselines. Notably, it achieves superior reconstruction naturalness while using up to 23% fewer tokens than a 40 Hz fixed-frame-rate baseline. VARSTok further yields lower word error rates and improved naturalness in zero-shot text-to-speech synthesis. To the best of our knowledge, this is the first work to demonstrate that a fully dynamic, variable-frame-rate acoustic speech tokenizer can be seamlessly integrated into downstream speech language models.

Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding

TL;DR

VARSTok tackles the mismatch between speech information density and fixed-token-rate tokenizers by introducing a fully dynamic, variable-frame-rate tokenizer that uses a temporal-aware density peak clustering algorithm and an implicit duration coding scheme to embed both content and duration into a single token ID. A single-codebook VQ framework, expanded token vocabulary, and an end-to-end training objective enable efficient, duration-aware tokens that can be directly used by autoregressive speech language models without extra duration predictors. Across LibriTTS reconstruction, ARCH semantic evaluation, and zero-shot TTS tasks, VARSTok achieves competitive or superior performance at lower average token rates (around Hz) compared with fixed-rate baselines, while improving perceptual naturalness and maintaining low WER. This work demonstrates that fully dynamic, duration-aware acoustic tokenization can be seamlessly integrated into downstream speech LMs, offering token efficiency and improved prosody without architectural changes.”

Abstract

Existing speech tokenizers typically assign a fixed number of tokens per second, regardless of the varying information density or temporal fluctuations in the speech signal. This uniform token allocation mismatches the intrinsic structure of speech, where information is distributed unevenly over time. To address this, we propose VARSTok, a VAriable-frame-Rate Speech Tokenizer that adapts token allocation based on local feature similarity. VARSTok introduces two key innovations: (1) a temporal-aware density peak clustering algorithm that adaptively segments speech into variable-length units, and (2) a novel implicit duration coding scheme that embeds both content and temporal span into a single token index, eliminating the need for auxiliary duration predictors. Extensive experiments show that VARSTok significantly outperforms strong fixed-rate baselines. Notably, it achieves superior reconstruction naturalness while using up to 23% fewer tokens than a 40 Hz fixed-frame-rate baseline. VARSTok further yields lower word error rates and improved naturalness in zero-shot text-to-speech synthesis. To the best of our knowledge, this is the first work to demonstrate that a fully dynamic, variable-frame-rate acoustic speech tokenizer can be seamlessly integrated into downstream speech language models.

Paper Structure

This paper contains 34 sections, 12 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Tokenization
  4. Adaptive Compression in Speech
  5. Proposed Methods
  6. Overview
  7. Model Architecture
  8. Speech Encoder and Decoder
  9. VQ Module
  10. Temporal-Aware Density Peak Clustering
  11. Implicit Duration Coding via Extended Index
  12. Training Objective
  13. Experiments
  14. Experimental Setup
  15. Datasets and Tasks
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Overview of VARSTok. Input waveform is converted into frame-level embeddings via a speech encoder. Temporal-aware density peak clustering adaptively segments them into variable-length clusters based on similarity and temporal continuity. Each cluster is mean-pooled and quantized using a VQ codebook to produce a discrete token whose index encodes both content and duration (i.e., number of frames spanned). Each token embedding is expanded back to frame-level representations according to its duration and passed to the decoder for waveform reconstruction.
  • Figure 2: Token boundary visualization on two speech segments for three tokenizers : WavTokenizer (75 Hz), WavTokenizer (40 Hz), and VARSTok (30.95 Hz). Vertical red lines indicate token boundaries and token IDs are annotated above.
  • Figure 3: Token boundary visualization on two speech segments for wavtokenizer and VARSTok. Vertical red lines indicate token boundaries and token IDs are annotated above.