Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QuarkAudio Technical Report

Chengwei Liu, Haoyin Yan, Shaofei Xue, Xiaotao Liang, Xiaofu Chen, Bin Gong, Zheng Xue, Gang Song

TL;DR

QuarkAudio addresses fragmentation in audio modeling by introducing a unified, decoder-only autoregressive framework. It combines a dual-stream H-Codec tokenizer with a unified audio language model that conditions on continuous semantic and acoustic features to generate discrete audio tokens, enabling diverse tasks through task-specific tokens. The H-Codec family extends frame-rate adaptability and high-fidelity 48 kHz audio while maintaining efficient generation, and the framework demonstrates competitive performance across SR, TSE, SS, VC, LASS, and editing tasks. This approach offers a scalable path toward a robust, instruction-driven audio foundation model with practical implications for multi-task audio processing and editing applications.

Abstract

Many existing audio processing and generation models rely on task-specific architectures, resulting in fragmented development efforts and limited extensibility. It is therefore promising to design a unified framework capable of handling multiple tasks, while providing robust instruction and audio understanding and high-quality audio generation. This requires a compatible paradigm design, a powerful backbone, and a high-fidelity audio reconstruction module. To meet these requirements, this technical report introduces QuarkAudio, a decoder-only autoregressive (AR) LM-based generative framework that unifies multiple tasks. The framework includes a unified discrete audio tokenizer, H-Codec, which incorporates self-supervised learning (SSL) representations into the tokenization and reconstruction process. We further propose several improvements to H-Codec, such as a dynamic frame-rate mechanism and extending the audio sampling rate to 48 kHz. QuarkAudio unifies tasks by using task-specific conditional information as the conditioning sequence of the decoder-only LM, and predicting discrete target audio tokens in an AR manner. The framework supports a wide range of audio processing and generation tasks, including speech restoration (SR), target speaker extraction (TSE), speech separation (SS), voice conversion (VC), and language-queried audio source separation (LASS). In addition, we extend downstream tasks to universal free-form audio editing guided by natural language instructions (including speech semantic editing and audio event editing). Experimental results show that H-Codec achieves high-quality audio reconstruction with a low frame rate, improving both the efficiency and performance of downstream audio generation, and that QuarkAudio delivers competitive or comparable performance to state-of-the-art task-specific or multi-task systems across multiple tasks.

QuarkAudio Technical Report

TL;DR

QuarkAudio addresses fragmentation in audio modeling by introducing a unified, decoder-only autoregressive framework. It combines a dual-stream H-Codec tokenizer with a unified audio language model that conditions on continuous semantic and acoustic features to generate discrete audio tokens, enabling diverse tasks through task-specific tokens. The H-Codec family extends frame-rate adaptability and high-fidelity 48 kHz audio while maintaining efficient generation, and the framework demonstrates competitive performance across SR, TSE, SS, VC, LASS, and editing tasks. This approach offers a scalable path toward a robust, instruction-driven audio foundation model with practical implications for multi-task audio processing and editing applications.

Abstract

Many existing audio processing and generation models rely on task-specific architectures, resulting in fragmented development efforts and limited extensibility. It is therefore promising to design a unified framework capable of handling multiple tasks, while providing robust instruction and audio understanding and high-quality audio generation. This requires a compatible paradigm design, a powerful backbone, and a high-fidelity audio reconstruction module. To meet these requirements, this technical report introduces QuarkAudio, a decoder-only autoregressive (AR) LM-based generative framework that unifies multiple tasks. The framework includes a unified discrete audio tokenizer, H-Codec, which incorporates self-supervised learning (SSL) representations into the tokenization and reconstruction process. We further propose several improvements to H-Codec, such as a dynamic frame-rate mechanism and extending the audio sampling rate to 48 kHz. QuarkAudio unifies tasks by using task-specific conditional information as the conditioning sequence of the decoder-only LM, and predicting discrete target audio tokens in an AR manner. The framework supports a wide range of audio processing and generation tasks, including speech restoration (SR), target speaker extraction (TSE), speech separation (SS), voice conversion (VC), and language-queried audio source separation (LASS). In addition, we extend downstream tasks to universal free-form audio editing guided by natural language instructions (including speech semantic editing and audio event editing). Experimental results show that H-Codec achieves high-quality audio reconstruction with a low frame rate, improving both the efficiency and performance of downstream audio generation, and that QuarkAudio delivers competitive or comparable performance to state-of-the-art task-specific or multi-task systems across multiple tasks.
Paper Structure (22 sections, 2 equations, 3 figures, 8 tables)

This paper contains 22 sections, 2 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Model Architecture
  3. Unified Discrete Tokenizer
  4. Encoder and Decoder
  5. Dynamic Frame Aggregation and De-aggregation
  6. Optimization Strategy
  7. Unified Audio Language Model
  8. Overall Framework
  9. Unifying Tasks with Operational Modes
  10. Experiments
  11. H-Codec
  12. Experimental Setup
  13. Experimental Results
  14. Unified Audio Generation
  15. Audio Processing Performance
  16. ...and 7 more sections

Figures (3)

  • Figure 1: The overall architecture of QuarkAudio, which is a straightforward model for multiple audio tasks. For simplicity, we illustrate the AR process with single-layer codec tokens and it actually operates in a multi-layer AR manner with delay pattern.
  • Figure 2: Architecture of H-Codec
  • Figure 3: H-Codec-2.0 training pipeline