Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LL-SDR: Low-Latency Speech enhancement through Discrete Representations

Jingyi Li, Luca Della Libera, Mirco Ravanelli, Cem Subakan

Abstract

Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autoregressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches, while enabling lightweight, low-latency speech enhancement in both reverberant and non-reverberant noisy environments. Demos and source code are available at our project websites.

LL-SDR: Low-Latency Speech enhancement through Discrete Representations

Abstract

Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autoregressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches, while enabling lightweight, low-latency speech enhancement in both reverberant and non-reverberant noisy environments. Demos and source code are available at our project websites.
Paper Structure (15 sections, 6 equations, 1 figure, 4 tables, 1 algorithm)

This paper contains 15 sections, 6 equations, 1 figure, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Framework Overview
  4. Variance-Ordered Residual Vector Quantizer
  5. Semantic Discriminator
  6. Training Details
  7. Experiments
  8. Dataset
  9. Metrics
  10. Baselines
  11. Results and Discussion
  12. Disentanglement Analysis
  13. Ablation Study
  14. Conclusion
  15. Generative AI Use Disclosure

Figures (1)

  • Figure 1: LL-SDR speech enhancement framework: a noisy waveform is encoded and enhanced in the discrete space via our variance-ordered quantizer where $N_e$ is four and $N_n$ is one, then decoded back to waveform. The triangle structure here illustrates that different quantizers have different dimensional sizes; a HuBERT-based semantic discriminator to align the enhanced representation with clean speech semantics.