Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast-ULCNet: A fast and ultra low complexity network for single-channel speech enhancement

Nicolás Arrieta Larraza, Niels de Koeijer

TL;DR

This work tackles the challenge of real-time, single-channel speech enhancement on resource-constrained devices by adapting the state-of-the-art ULCNet architecture. It replaces GRU layers with FastGRNN units to reduce parameters and latency and introduces Comfi-FastGRNN, a trainable complementary filter that mitigates RNN state drift over long sequences. The resulting Fast-ULCNet matches ULCNet in enhancement quality while halving parameter counts and achieving ~34% lower latency, with Comfi-FastGRNN preserving performance on longer audio segments. Evaluations on the DNS Challenge data using DNSMOS, PESQ, and SI-SDR demonstrate these improvements, and the authors provide open-source implementations and a demo to facilitate deployment in real-time systems.

Abstract

Single-channel speech enhancement algorithms are often used in resource-constrained embedded devices, where low latency and low complexity designs gain more importance. In recent years, researchers have proposed a wide variety of novel solutions to this problem. In particular, a recent deep learning model named ULCNet is among the state-of-the-art approaches in this domain. This paper proposes an adaptation of ULCNet, by replacing its GRU layers with FastGRNNs, to reduce both computational latency and complexity. Furthermore, this paper shows empirical evidence on the performance decay of FastGRNNs in long audio signals during inference due to internal state drifting, and proposes a novel approach based on a trainable complementary filter to mitigate it. The resulting model, Fast-ULCNet, performs on par with the state-of-the-art original ULCNet architecture on a speech enhancement task, while reducing its model size by more than half and decreasing its latency by 34% on average.

Fast-ULCNet: A fast and ultra low complexity network for single-channel speech enhancement

TL;DR

This work tackles the challenge of real-time, single-channel speech enhancement on resource-constrained devices by adapting the state-of-the-art ULCNet architecture. It replaces GRU layers with FastGRNN units to reduce parameters and latency and introduces Comfi-FastGRNN, a trainable complementary filter that mitigates RNN state drift over long sequences. The resulting Fast-ULCNet matches ULCNet in enhancement quality while halving parameter counts and achieving ~34% lower latency, with Comfi-FastGRNN preserving performance on longer audio segments. Evaluations on the DNS Challenge data using DNSMOS, PESQ, and SI-SDR demonstrate these improvements, and the authors provide open-source implementations and a demo to facilitate deployment in real-time systems.

Abstract

Single-channel speech enhancement algorithms are often used in resource-constrained embedded devices, where low latency and low complexity designs gain more importance. In recent years, researchers have proposed a wide variety of novel solutions to this problem. In particular, a recent deep learning model named ULCNet is among the state-of-the-art approaches in this domain. This paper proposes an adaptation of ULCNet, by replacing its GRU layers with FastGRNNs, to reduce both computational latency and complexity. Furthermore, this paper shows empirical evidence on the performance decay of FastGRNNs in long audio signals during inference due to internal state drifting, and proposes a novel approach based on a trainable complementary filter to mitigate it. The resulting model, Fast-ULCNet, performs on par with the state-of-the-art original ULCNet architecture on a speech enhancement task, while reducing its model size by more than half and decreasing its latency by 34% on average.
Paper Structure (14 sections, 3 equations, 3 figures, 2 tables)

This paper contains 14 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Fast-ULCNet
  3. FastGRNN and Comfi-FastGRNN
  4. Model architecture
  5. Experiments
  6. Implementation details
  7. Architecture implementation
  8. Loss function
  9. Dataset
  10. Training setup
  11. Results
  12. Objective evaluation
  13. Computational complexity
  14. Conclusion

Figures (3)

  • Figure 1: Block diagram of Comfi-FastGRNN, comprising the original FastGRNN architecture extended with a trainable complementary filter.
  • Figure 2: Fast-ULCNet inference shows drifting on the mean RNN state ${h}_t$ (top) and performance decay on the processed signal (bottom) over time with FastGRNN (left column), whereas Comfi-FastGRNN (right column) maintains stable mean RNN state $h_{t\mathrm{comfi}}$ and consistent performance.
  • Figure 3: Architecture of Fast-ULCNet. Black boxes represent components from the original ULCNet architecture, while dotted light-blue boxes highlight the FastGRNN-based modifications introduced in this work, with or without the complementary filter. Subscripts of $X$ (phase, mag, re, and im) indicate the phase, magnitude, real, and imaginary parts of the input features, respectively.