Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speech Enhancement Using Continuous Embeddings of Neural Audio Codec

Haoyang Li, Jia Qi Yip, Tianyu Fan, Eng Siong Chng

TL;DR

This work tackles efficient speech enhancement for cloud-based audio transmission by performing SE directly in the continuous embedding space of a pretrained neural audio codec (NAC), avoiding computationally heavy LM-based token prediction. It introduces an embedding-domain SE model guided by an embedding loss and supplementary time- and frequency-domain losses, enabling fast, compressor-friendly enhancement on NAC embeddings. The method achieves a real-time factor of $0.005$ and around $3.94$ GMACs, roughly an $18\times$ reduction in MACs compared with Sepformer, while delivering competitive speech quality on DNS challenge test sets. This work demonstrates the practicality of NAC-based SE for codec-first cloud pipelines and points toward scalable, low-latency speech enhancement in streaming architectures.

Abstract

Recent advancements in Neural Audio Codec (NAC) models have inspired their use in various speech processing tasks, including speech enhancement (SE). In this work, we propose a novel, efficient SE approach by leveraging the pre-quantization output of a pretrained NAC encoder. Unlike prior NAC-based SE methods, which process discrete speech tokens using Language Models (LMs), we perform SE within the continuous embedding space of the pretrained NAC, which is highly compressed along the time dimension for efficient representation. Our lightweight SE model, optimized through an embedding-level loss, delivers results comparable to SE baselines trained on larger datasets, with a significantly lower real-time factor of 0.005. Additionally, our method achieves a low GMAC of 3.94, reducing complexity 18-fold compared to Sepformer in a simulated cloud-based audio transmission environment. This work highlights a new, efficient NAC-based SE solution, particularly suitable for cloud applications where NAC is used to compress audio before transmission. Copyright 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Speech Enhancement Using Continuous Embeddings of Neural Audio Codec

TL;DR

This work tackles efficient speech enhancement for cloud-based audio transmission by performing SE directly in the continuous embedding space of a pretrained neural audio codec (NAC), avoiding computationally heavy LM-based token prediction. It introduces an embedding-domain SE model guided by an embedding loss and supplementary time- and frequency-domain losses, enabling fast, compressor-friendly enhancement on NAC embeddings. The method achieves a real-time factor of and around GMACs, roughly an reduction in MACs compared with Sepformer, while delivering competitive speech quality on DNS challenge test sets. This work demonstrates the practicality of NAC-based SE for codec-first cloud pipelines and points toward scalable, low-latency speech enhancement in streaming architectures.

Abstract

Recent advancements in Neural Audio Codec (NAC) models have inspired their use in various speech processing tasks, including speech enhancement (SE). In this work, we propose a novel, efficient SE approach by leveraging the pre-quantization output of a pretrained NAC encoder. Unlike prior NAC-based SE methods, which process discrete speech tokens using Language Models (LMs), we perform SE within the continuous embedding space of the pretrained NAC, which is highly compressed along the time dimension for efficient representation. Our lightweight SE model, optimized through an embedding-level loss, delivers results comparable to SE baselines trained on larger datasets, with a significantly lower real-time factor of 0.005. Additionally, our method achieves a low GMAC of 3.94, reducing complexity 18-fold compared to Sepformer in a simulated cloud-based audio transmission environment. This work highlights a new, efficient NAC-based SE solution, particularly suitable for cloud applications where NAC is used to compress audio before transmission. Copyright 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Paper Structure

This paper contains 17 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overall Architecture
  4. Neural Audio Codec
  5. SE Module
  6. Loss Functions
  7. Embedding Loss Function
  8. Time and Frequency domain Loss Functions
  9. Overall Loss Function
  10. Experiments
  11. Dataset and Evaluation Metrics
  12. Experimental Setup and Baselines
  13. Results and discussions
  14. Comparison with baselines
  15. Investigation on loss functions
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Overview of different speech enhancement pipelines depicting scenarios where speech is processed on the cloud and transmitted to a local device. (a) Our approach (b) Previous NAC-based SE solutions leveraging LM wang2024speechxyang2024genhancer (c) SE methods that do not utilize NAC
  • Figure 2: (a) Overall architecture of our SE pipeline (b) Architecture of our embedding-domain SE model