Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization

Luca Della Libera, Cem Subakan, Mirco Ravanelli

TL;DR

DyCAST tackles inefficiencies in speech tokenization by introducing dynamic, character-aligned tokens with soft alignment and explicit duration modeling, enabling variable-frame-rate representations. The framework integrates a hazard-based dynamic chunking mechanism, a negative-binomial duration model, and a retrieval-augmented decoding strategy to preserve detail at low frame rates without increasing bitrate. Across speech resynthesis, voice conversion, ASR/SI/SER probing, and TTS, DyCAST delivers competitive quality while using substantially fewer tokens than fixed-rate codecs. This approach offers practical advantages for downstream multimodal modeling by enabling content-adaptive frame rates and flexible bitrate-timing trade-offs, with the retrieval mechanism further enhancing reconstruction fidelity in data-limited scenarios.

Abstract

Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete tokens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. To improve speech resynthesis quality at low frame rates, we further introduce a retrieval-augmented decoding mechanism that enhances reconstruction fidelity without increasing bitrate. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs.

Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization

TL;DR

DyCAST tackles inefficiencies in speech tokenization by introducing dynamic, character-aligned tokens with soft alignment and explicit duration modeling, enabling variable-frame-rate representations. The framework integrates a hazard-based dynamic chunking mechanism, a negative-binomial duration model, and a retrieval-augmented decoding strategy to preserve detail at low frame rates without increasing bitrate. Across speech resynthesis, voice conversion, ASR/SI/SER probing, and TTS, DyCAST delivers competitive quality while using substantially fewer tokens than fixed-rate codecs. This approach offers practical advantages for downstream multimodal modeling by enabling content-adaptive frame rates and flexible bitrate-timing trade-offs, with the retrieval mechanism further enhancing reconstruction fidelity in data-limited scenarios.

Abstract

Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete tokens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. To improve speech resynthesis quality at low frame rates, we further introduce a retrieval-augmented decoding mechanism that enhances reconstruction fidelity without increasing bitrate. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs.
Paper Structure (47 sections, 9 equations, 3 figures, 6 tables)

This paper contains 47 sections, 9 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fixed-Frame-Rate Codecs.
  4. Variable-Frame-Rate Codecs.
  5. Text-Aligned Speech Representations.
  6. DyCAST
  7. Dynamic Chunking
  8. Dechunking
  9. Retrieval-Augmented Decoding
  10. Experimental Setup
  11. Architecture
  12. Character Aligner.
  13. Encoder and Decoder.
  14. Compressor and Decompressor.
  15. Boundary and Duration Predictor.
  16. ...and 32 more sections

Figures (3)

  • Figure 1: DyCAST architecture. Frame-level representations extracted by a frozen, self-supervised encoder are compressed and dynamically grouped into variable-length chunks, pooled, and quantized into discrete tokens. The decoding stage reverses this process by expanding token-level representations back to frame-level features before waveform reconstruction. Character-level boundaries are provided during training by a frozen aligner.
  • Figure 2: Retrieval-augmented decoding (RAD). Discrete latents are refined via similarity search against a pool of continuous latents prior to waveform reconstruction.
  • Figure 3: DyCAST chunk boundaries at average frame rates of approximately 14 Hz, 17 Hz, 9 Hz, and 6 Hz (from left to right).