Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DashengTokenizer: One layer is enough for unified audio understanding and generation

Heinrich Dinkel, Xingwei Sun, Gang Li, Jiahao Mei, Yadong Niu, Jizhong Liu, Xiyang Li, Yifan Liao, Jiahao Zhou, Junbo Zhang, Jian Luan

TL;DR

The DashengTokenizer is introduced, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks that surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder.

Abstract

This paper introduces DashengTokenizer, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks. Unlike conventional approaches, which train acoustic tokenizers and subsequently integrate frozen semantic knowledge, our method inverts this paradigm: we leverage frozen semantic features and inject acoustic information. In linear evaluation across 22 diverse tasks, our method outperforms previous audio codec and audio encoder baselines by a significant margin while maintaining competitive audio reconstruction quality. Notably, we demonstrate that this acoustic injection improves performance for tasks such as speech emotion recognition, music understanding, and acoustic scene classification. We further evaluate the tokenizer's generative performance on text-to-audio (TTA), text-to-music (TTM), and speech enhancement (SE). Our approach surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder. Finally, our results challenge the prevailing assumption that VAE-based architectures are a prerequisite for audio synthesis. Checkpoints are available at https://huggingface.co/mispeech/dashengtokenizer.

DashengTokenizer: One layer is enough for unified audio understanding and generation

TL;DR

The DashengTokenizer is introduced, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks that surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder.

Abstract

This paper introduces DashengTokenizer, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks. Unlike conventional approaches, which train acoustic tokenizers and subsequently integrate frozen semantic knowledge, our method inverts this paradigm: we leverage frozen semantic features and inject acoustic information. In linear evaluation across 22 diverse tasks, our method outperforms previous audio codec and audio encoder baselines by a significant margin while maintaining competitive audio reconstruction quality. Notably, we demonstrate that this acoustic injection improves performance for tasks such as speech emotion recognition, music understanding, and acoustic scene classification. We further evaluate the tokenizer's generative performance on text-to-audio (TTA), text-to-music (TTM), and speech enhancement (SE). Our approach surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder. Finally, our results challenge the prevailing assumption that VAE-based architectures are a prerequisite for audio synthesis. Checkpoints are available at https://huggingface.co/mispeech/dashengtokenizer.
Paper Structure (23 sections, 5 equations, 3 figures, 11 tables)

This paper contains 23 sections, 5 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Previous works
  3. Codecs
  4. Unified tokenizers
  5. Methodology
  6. Experiments
  7. Datasets
  8. Evaluation benchmarks
  9. Setup
  10. Architecture Configuration
  11. Semantic encoder
  12. Acoustic Encoder
  13. Acoustic decoder
  14. Results
  15. Reconstruction evaluation
  16. ...and 8 more sections

Figures (3)

  • Figure 1: A summarization of DashengTokenizer capabilities for understanding and generation tasks.
  • Figure 2: The proposed DashengTokenizer compared to prior approaches: [A] standard acoustic Acoustic modeling using VAE, and [B] semantically distilled (VQ-)VAEs. In contrast, our approach eliminates the multi-stage training required by [B] and does not rely on a semantic decoder that is discarded during inference, thereby avoiding a train-test mismatch.
  • Figure 3: Text-to-Audio and Text-to-Music training progress of our proposed framework compared with the VAE from UniFlow-Audio.