Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Speech Enhancement with Dynamic Channel Pruning

Riccardo Miccini, Clement Laroche, Tobias Piechowiak, Luca Pezzarossa

TL;DR

This paper introduces Dynamic Channel Pruning (DynCP) for speech enhancement by integrating a gating module that selectively activates convolutional channels at runtime, enabling adaptive computation on edge devices. The authors design Conv-FSENet, a fully convolutional architecture with a front-end, a dilated depthwise-separable conv-based TCN backbone, and a back-end that generates a denoising mask, while DynCP enables channel-wise sparsity via binary gates guided by a target pruning ratio $\Phi_{trgt}$. They train with surrogate gradients and an auxiliary loss $\mathcal{L}_{DCP}$ to enforce the desired channel usage, achieving up to $29.6\%$ MAC savings with only a $0.75\%$ PESQ drop on VoiceBank+DEMAND data, and analyze the trade-offs between causality, backprop methods, and dynamism. The work demonstrates a practical path toward deploying more powerful SE models on resource-constrained devices by making computation scale with input difficulty and runtime conditions.

Abstract

Speech Enhancement (SE) is essential for improving productivity in remote collaborative environments. Although deep learning models are highly effective at SE, their computational demands make them impractical for embedded systems. Furthermore, acoustic conditions can change significantly in terms of difficulty, whereas neural networks are usually static with regard to the amount of computation performed. To this end, we introduce Dynamic Channel Pruning to the audio domain for the first time and apply it to a custom convolutional architecture for SE. Our approach works by identifying unnecessary convolutional channels at runtime and saving computational resources by not computing the activations for these channels and retrieving their filters. When trained to only use 25% of channels, we save 29.6% of MACs while only causing a 0.75% drop in PESQ. Thus, DynCP offers a promising path toward deploying larger and more powerful SE solutions on resource-constrained devices.

Scalable Speech Enhancement with Dynamic Channel Pruning

TL;DR

This paper introduces Dynamic Channel Pruning (DynCP) for speech enhancement by integrating a gating module that selectively activates convolutional channels at runtime, enabling adaptive computation on edge devices. The authors design Conv-FSENet, a fully convolutional architecture with a front-end, a dilated depthwise-separable conv-based TCN backbone, and a back-end that generates a denoising mask, while DynCP enables channel-wise sparsity via binary gates guided by a target pruning ratio . They train with surrogate gradients and an auxiliary loss to enforce the desired channel usage, achieving up to MAC savings with only a PESQ drop on VoiceBank+DEMAND data, and analyze the trade-offs between causality, backprop methods, and dynamism. The work demonstrates a practical path toward deploying more powerful SE models on resource-constrained devices by making computation scale with input difficulty and runtime conditions.

Abstract

Speech Enhancement (SE) is essential for improving productivity in remote collaborative environments. Although deep learning models are highly effective at SE, their computational demands make them impractical for embedded systems. Furthermore, acoustic conditions can change significantly in terms of difficulty, whereas neural networks are usually static with regard to the amount of computation performed. To this end, we introduce Dynamic Channel Pruning to the audio domain for the first time and apply it to a custom convolutional architecture for SE. Our approach works by identifying unnecessary convolutional channels at runtime and saving computational resources by not computing the activations for these channels and retrieving their filters. When trained to only use 25% of channels, we save 29.6% of MACs while only causing a 0.75% drop in PESQ. Thus, DynCP offers a promising path toward deploying larger and more powerful SE solutions on resource-constrained devices.

Paper Structure

This paper contains 9 sections, 5 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Conv-FSENet Architecture
  4. Problem formulation
  5. Model Architecture
  6. Dynamic Channel Pruning
  7. Experimental setup
  8. Results
  9. Conclusion

Figures (4)

  • Figure 1: Noisy input, channel states, and predicted speech computed from an 8 sample; x-axis shows time, y-axis shows frequency for spectrograms and channel index for $\mathcal{B}_i$; white channels are kept, black channels are omitted; green lines and titles indicate the instantaneous and average pruning ratio for the given block, respectively.
  • Figure 2: Overall architecture of Conv-FSENet.
  • Figure 3: Structure of baseline block $\mathcal{B}_i$ (red box) and gating module $\mathcal{G}_i$ (purple box) for DynCP; annotations along the graph refer to activation shapes (right: training, left: streaming inference), while those inside the convolutional layers refer to weights.
  • Figure 4: Pareto fronts of PESQ vs. MACs for static baselines and DynCP models over a range of $N_s$ and $\Phi_\text{trgt}$ values.