Adaptive Slimming for Scalable and Efficient Speech Enhancement

TL;DR

This work tackles the challenge of resource-efficient speech enhancement on edge devices by introducing dynamic slimming to the DEMUCS architecture. A lightweight routing sub-network selects how much of the model to used (UF) per utterance, balancing performance and computational cost with end-to-end training that leverages Gumbel-softmax. Empirical results on VoiceBank+DEMAND show Pareto-optimal trade-offs, including substantial MAC reductions with equal or improved speech quality, and clear adaptation of resources to input difficulty. The approach enables scalable, real-time SE suitable for constrained environments and offers a path for extending adaptive routing to other SE models.

Abstract

Speech enhancement (SE) enables robust speech recognition, real-time communication, hearing aids, and other applications where speech quality is crucial. However, deploying such systems on resource-constrained devices involves choosing a static trade-off between performance and computational efficiency. In this paper, we introduce dynamic slimming to DEMUCS, a popular SE architecture, making it scalable and input-adaptive. Slimming lets the model operate at different utilization factors (UF), each corresponding to a different performance/efficiency trade-off, effectively mimicking multiple model sizes without the extra storage costs. In addition, a router subnet, trained end-to-end with the backbone, determines the optimal UF for the current input. Thus, the system saves resources by adaptively selecting smaller UFs when additional complexity is unnecessary. We show that our solution is Pareto-optimal against individual UFs, confirming the benefits of dynamic routing. When training the proposed dynamically-slimmable model to use 10% of its capacity on average, we obtain the same or better speech quality as the equivalent static 25% utilization while reducing MACs by 29%.

Figures (6)

• Figure 1: Relationship between average utilization and SI-SDR improvement for model trained with $\upsilon_{\text{trgt}} = 0.5$, showing how our proposed solution processes noisier input data (lighter points) using more resources, causing a larger improvement.
• Figure 2: Overall solution showing DEMUCS backbone with encoder $\mathcal{E}_i$ and decoder $\mathcal{D}_i$ blocks (gray), the grouped GRU bottleneck (orange), and routing subnet $\mathcal{R}$ (red). Connections between blocks are annotated with signal dimensionality. Dotted connections represent UF arguments.
• Figure 3: Slimmable blocks with weight tensors; width, height, and depth correspond to input channels, output channels, and kernel size, respectively; $C = 2^{i-2}H$ for $i \ge 2$ is the current hidden size.
• Figure 4: Example inference for models trained with $\upsilon_{\text{trgt}} \in \left\{ 0.25, 0.5, 0.75 \right\}$, showing input spectrogram along with improvement in instantaneous SNR (orange line, with yellow regions marking difference with preceding model on the left) and chosen UFs over time. In the middle portion of the sample (in green), we deliberately reduce the noise level by 10 to show how each model reacts by decreasing its average utilization.
• Figure 5: Pareto fronts given by utilization/MACs (x-axes) vs. speech quality metrics (y-axis) for static slimmable backbone (light blue empty squares) and dynamic models trained on a range of $\upsilon_{\text{trgt}}$; the dotted horizontal lines show the distance between each $\upsilon_{\text{trgt}}$ (solid vertical lines) and actual average utilization. For $\upsilon_{\text{trgt}} \in \left\{0.25, 0.5 \right\}$, we include results from the alternative training schedule described in \ref{['itm:exp_training']}.