Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SepPrune: Structured Pruning for Efficient Deep Speech Separation

Yuqi Li, Kai Li, Xin Yin, Zhifei Yang, Junhao Dong, Zeyu Dong, Chuanguang Yang, Yingli Tian, Yao Lu

TL;DR

SepPrune tackles the critical mismatch between high separation quality and real-time efficiency in deep speech separation. It introduces a structured channel pruning approach tailored to end-to-end models, combining structural analysis, differentiable masks learned via Gumbel-Softmax with a refined Straight-Through estimator, and subsequent pruning with weight refinement. Across Libri2Mix, LRS2-2Mix, and EchoSet, SepPrune consistently outperforms prior pruning baselines and can recover over $85 ext{%}$ of the original SDRi/SI-SDRi with just one epoch of fine-tuning, while achieving up to $36\times$ faster convergence than training from scratch. The work demonstrates meaningful practical speedups and memory savings, enabling deployment of sophisticated speech separation systems on resource-constrained devices. It also highlights limitations in testing on the latest state-of-the-art models and outlines future work to broaden generalization to more architectures such as Tiger and SPMamba.

Abstract

Although deep learning has substantially advanced speech separation in recent years, most existing studies continue to prioritize separation quality while overlooking computational efficiency, an essential factor for low-latency speech processing in real-time applications. In this paper, we propose SepPrune, the first structured pruning framework specifically designed to compress deep speech separation models and reduce their computational cost. SepPrune begins by analyzing the computational structure of a given model to identify layers with the highest computational burden. It then introduces a differentiable masking strategy to enable gradient-driven channel selection. Based on the learned masks, SepPrune prunes redundant channels and fine-tunes the remaining parameters to recover performance. Extensive experiments demonstrate that this learnable pruning paradigm yields substantial advantages for channel pruning in speech separation models, outperforming existing methods. Notably, a model pruned with SepPrune can recover 85% of the performance of a pre-trained model (trained over hundreds of epochs) with only one epoch of fine-tuning, and achieves convergence 36$\times$ faster than training from scratch. Code is available at https://github.com/itsnotacie/SepPrune.

SepPrune: Structured Pruning for Efficient Deep Speech Separation

TL;DR

SepPrune tackles the critical mismatch between high separation quality and real-time efficiency in deep speech separation. It introduces a structured channel pruning approach tailored to end-to-end models, combining structural analysis, differentiable masks learned via Gumbel-Softmax with a refined Straight-Through estimator, and subsequent pruning with weight refinement. Across Libri2Mix, LRS2-2Mix, and EchoSet, SepPrune consistently outperforms prior pruning baselines and can recover over of the original SDRi/SI-SDRi with just one epoch of fine-tuning, while achieving up to faster convergence than training from scratch. The work demonstrates meaningful practical speedups and memory savings, enabling deployment of sophisticated speech separation systems on resource-constrained devices. It also highlights limitations in testing on the latest state-of-the-art models and outlines future work to broaden generalization to more architectures such as Tiger and SPMamba.

Abstract

Although deep learning has substantially advanced speech separation in recent years, most existing studies continue to prioritize separation quality while overlooking computational efficiency, an essential factor for low-latency speech processing in real-time applications. In this paper, we propose SepPrune, the first structured pruning framework specifically designed to compress deep speech separation models and reduce their computational cost. SepPrune begins by analyzing the computational structure of a given model to identify layers with the highest computational burden. It then introduces a differentiable masking strategy to enable gradient-driven channel selection. Based on the learned masks, SepPrune prunes redundant channels and fine-tunes the remaining parameters to recover performance. Extensive experiments demonstrate that this learnable pruning paradigm yields substantial advantages for channel pruning in speech separation models, outperforming existing methods. Notably, a model pruned with SepPrune can recover 85% of the performance of a pre-trained model (trained over hundreds of epochs) with only one epoch of fine-tuning, and achieves convergence 36 faster than training from scratch. Code is available at https://github.com/itsnotacie/SepPrune.
Paper Structure (17 sections, 8 equations, 2 figures, 8 tables)

This paper contains 17 sections, 8 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Model Pruning
  4. Speech Separation
  5. Efficient Speech Separation
  6. Preliminary
  7. Method
  8. Compressing Speech Separation Models via Channel Pruning
  9. SepPrune: Structured Channel Pruning via Differentiable Masks
  10. Experiments
  11. Model and Dataset
  12. Training and Evaluation
  13. Comparisons with State-of-The-Art Methods
  14. Separation Effciency
  15. SepPrune Enables Fast Convergence
  16. ...and 2 more sections

Figures (2)

  • Figure 1: The overall pipeline of the proposed SepPrune.
  • Figure 2: Visualization of the obtained channel masks on the LRS2-2Mix dataset. For the convenience of visualization, we select the first layer on A-FRCNN-12, SuDoRM-RF1.0x and TDANet for visualization, and reshape the masks into $16\times32$, $16\times32$ and $32\times32$.