GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling

TL;DR

GenSE addresses speech enhancement under noise by leveraging semantic information via language models. It reframes SE as a conditional language-modeling task using discrete semantic tokens and a single-quantizer acoustic-token codec, organized in a two-stage hierarchical framework (N2S and S2S) with token-chain prompting to preserve timbre. The paper introduces SimCodec, a single-quantizer neural codec with a codebook reorganization that achieves high reconstruction quality with fewer tokens, and demonstrates that GenSE surpasses state-of-the-art SE methods on DNS and CHiME-4 in both objective and subjective measures. Ablation studies confirm the importance of hierarchical modeling and token-chain prompting, and results on SimCodec show efficient token usage and robust reconstruction. These advances indicate a scalable, semantically informed approach to robust speech enhancement with strong generalization.

Abstract

Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, we employ language models as an efficient semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called \textit{GenSE}. Specifically, we approach SE as a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using a pre-trained self-supervised model and into acoustic tokens using a custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose a hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce a token chain prompting mechanism during the acoustic token generation stage to ensure timbre consistency throughout the speech enhancement process. Experimental results on benchmark datasets demonstrate that our proposed approach outperforms state-of-the-art SE systems in terms of speech quality and generalization capability.

Figures (7)

• Figure 1: The hierarchical modeling framework of language model in GenSE.
• Figure 2: The relationship between reconstruction quality and codebook usage across different codebook sizes.
• Figure 3: The detailed architecture and training process of SimCodec, with the reorganization process of the group quantizer is highlighted in the red dashed block.
• Figure 4: Violin plots for speech naturalness and speaker similarity, comparing the signals enhanced by baseline systems and GenSE. The thin inner line of each violin plot represents the full range of MOS values, while the thick inner line marks the interquartile range (between the first and third quartiles). The white point in each plot indicates the median value.
• Figure 5: ABX results between GenSE and baseline systems. "No Preference" means that participants perceive the enhanced signal to be as realistic as the clean signal.