Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Papez: Resource-Efficient Speech Separation with Auditory Working Memory

Hyunseok Oh, Juheon Yi, Youngki Lee

TL;DR

The paper tackles the heavy computational burden of transformer-based single-channel speech separation in mobile and IoT contexts. It introduces Papez, which replaces the inter-chunk transformer with a compact Auditory Working Memory Transformer and employs Adaptive Token Pruning to omit redundant tokens, aided by a recurrent transformer to share parameters. The approach yields substantial efficiency gains in parameters and latency while maintaining or improving separation accuracy on WSJ0-2Mix and LibriMix compared with strong baselines. The results show up to 17.7x model size reduction relative to Sepformer with minimal accuracy loss, and demonstrate the method's applicability to other transformer-based models, highlighting strong practical potential for real-time, resource-constrained deployments.

Abstract

Transformer-based models recently reached state-of-the-art single-channel speech separation accuracy; However, their extreme computational load makes it difficult to deploy them in resource-constrained mobile or IoT devices. We thus present Papez, a lightweight and computation-efficient single-channel speech separation model. Papez is based on three key techniques. We first replace the inter-chunk Transformer with small-sized auditory working memory. Second, we adaptively prune the input tokens that do not need further processing. Finally, we reduce the number of parameters through the recurrent transformer. Our extensive evaluation shows that Papez achieves the best resource and accuracy tradeoffs with a large margin. We publicly share our source code at \texttt{https://github.com/snuhcs/Papez}

Papez: Resource-Efficient Speech Separation with Auditory Working Memory

TL;DR

The paper tackles the heavy computational burden of transformer-based single-channel speech separation in mobile and IoT contexts. It introduces Papez, which replaces the inter-chunk transformer with a compact Auditory Working Memory Transformer and employs Adaptive Token Pruning to omit redundant tokens, aided by a recurrent transformer to share parameters. The approach yields substantial efficiency gains in parameters and latency while maintaining or improving separation accuracy on WSJ0-2Mix and LibriMix compared with strong baselines. The results show up to 17.7x model size reduction relative to Sepformer with minimal accuracy loss, and demonstrate the method's applicability to other transformer-based models, highlighting strong practical potential for real-time, resource-constrained deployments.

Abstract

Transformer-based models recently reached state-of-the-art single-channel speech separation accuracy; However, their extreme computational load makes it difficult to deploy them in resource-constrained mobile or IoT devices. We thus present Papez, a lightweight and computation-efficient single-channel speech separation model. Papez is based on three key techniques. We first replace the inter-chunk Transformer with small-sized auditory working memory. Second, we adaptively prune the input tokens that do not need further processing. Finally, we reduce the number of parameters through the recurrent transformer. Our extensive evaluation shows that Papez achieves the best resource and accuracy tradeoffs with a large margin. We publicly share our source code at \texttt{https://github.com/snuhcs/Papez}
Paper Structure (12 sections, 2 equations, 3 figures, 3 tables)

This paper contains 12 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Architecture Overview
  5. Auditory Working Memory (AWM) Transformer
  6. Adaptive Token Pruning
  7. Experiment
  8. Experimental Setup
  9. Comparison with Prior Works
  10. Ablation Study
  11. Conclusion
  12. Acknowledgement

Figures (3)

  • Figure 1: Overall architecture of our Papez model. The iteration steps of the AWM Transformer layer are determined by our Adaptive Token Pruning technique.
  • Figure 2: Redundancy of input tokens in the Sepformer's dual-path process. The black dashed line indicates the threshold $P_{th} = 0.9$ of adaptively pruning the redundant token.
  • Figure 3: Operation of Adaptive Token Pruning.