Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniSE: A Unified Framework for Decoder-only Autoregressive LM-based Speech Enhancement

Haoyin Yan, Chengwei Liu, Shaofei Xue, Xiaotao Liang, Zheng Xue

TL;DR

UniSE addresses the challenge of unifying multiple speech enhancement tasks by leveraging a decoder-only autoregressive LM conditioned on continuous features from a WavLM encoder and discrete target-speech tokens from a BiCodec. It introduces task tokens to switch among speech restoration, target speaker extraction, and speaker-separated modes, enabling a single model to perform SR, TSE, and SS through multi-task training. Experimental results across DNS, URGENT, Libri2Mix, and WSJ0-2mix demonstrate competitive performance against discriminative and generative baselines while utilizing a compact 63M-parameter backbone. This approach showcases the potential of autoregressive LMs for unified SE and points toward broader applications in audio processing with efficient codec-based representations.

Abstract

The development of neural audio codecs (NACs) has largely promoted applications of language models (LMs) to speech processing and understanding. However, there lacks the verification on the effectiveness of autoregressive (AR) LMbased models in unifying different sub-tasks of speech enhancement (SE). In this work, we propose UniSE, a unified decoder-only LM-based framework to handle different SE tasks including speech restoration, target speaker extraction and speech separation. It takes input speech features as conditions and generates discrete tokens of the target speech using AR modeling, which facilitates a compatibility between distinct learning patterns of multiple tasks. Experiments on several benchmarks indicate the proposed UniSE can achieve competitive performance compared to discriminative and generative baselines, showing the capacity of LMs in unifying SE tasks. The demo page is available here: https://github.com/hyyan2k/UniSE.

UniSE: A Unified Framework for Decoder-only Autoregressive LM-based Speech Enhancement

TL;DR

UniSE addresses the challenge of unifying multiple speech enhancement tasks by leveraging a decoder-only autoregressive LM conditioned on continuous features from a WavLM encoder and discrete target-speech tokens from a BiCodec. It introduces task tokens to switch among speech restoration, target speaker extraction, and speaker-separated modes, enabling a single model to perform SR, TSE, and SS through multi-task training. Experimental results across DNS, URGENT, Libri2Mix, and WSJ0-2mix demonstrate competitive performance against discriminative and generative baselines while utilizing a compact 63M-parameter backbone. This approach showcases the potential of autoregressive LMs for unified SE and points toward broader applications in audio processing with efficient codec-based representations.

Abstract

The development of neural audio codecs (NACs) has largely promoted applications of language models (LMs) to speech processing and understanding. However, there lacks the verification on the effectiveness of autoregressive (AR) LMbased models in unifying different sub-tasks of speech enhancement (SE). In this work, we propose UniSE, a unified decoder-only LM-based framework to handle different SE tasks including speech restoration, target speaker extraction and speech separation. It takes input speech features as conditions and generates discrete tokens of the target speech using AR modeling, which facilitates a compatibility between distinct learning patterns of multiple tasks. Experiments on several benchmarks indicate the proposed UniSE can achieve competitive performance compared to discriminative and generative baselines, showing the capacity of LMs in unifying SE tasks. The demo page is available here: https://github.com/hyyan2k/UniSE.
Paper Structure (10 sections, 2 equations, 1 figure, 6 tables)

This paper contains 10 sections, 2 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. METHODOLOGY
  3. Conditional Feature Extractor
  4. Discrete Token Codec
  5. Unified Multi-Task Framework
  6. Inference Strategies
  7. EXPERIMENTS
  8. Experimental Setup
  9. Experimental Results
  10. CONCLUSION

Figures (1)

  • Figure 1: Overall architecture of UniSE, where the BiCodec Encoder is only utilized to generate label tokens during training and excluded during inference. The snowflake icon means that parameters are pre-trained and frozen, and the fire icon indicates that parameters are optimized during training.