Causal Speech Enhancement with Predicting Semantics based on Quantized Self-supervised Learning Features
Emiru Tsunoo, Yuki Saito, Wataru Nakata, Hiroshi Saruwatari
TL;DR
This work tackles real-time speech enhancement under strict causality by leveraging self-supervised learning features and introducing semantic token prediction. The authors propose a one-stage model that quantizes causal SSL features into semantic tokens and concurrently predicts future tokens through multi-task learning, boosting performance. On the VoiceBank + DEMAND dataset, the method achieves a PESQ of 2.88, with semantic-prediction contributing a meaningful gain, and ablations highlight the importance of FiLM fusion and causal Transformer design. The approach offers a practical pathway for deploying SSL-informed, causal SE with token-based future prediction, suggesting directions for fully causal SSL training and improved phase handling in future work.
Abstract
Real-time speech enhancement (SE) is essential to online speech communication. Causal SE models use only the previous context while predicting future information, such as phoneme continuation, may help performing causal SE. The phonetic information is often represented by quantizing latent features of self-supervised learning (SSL) models. This work is the first to incorporate SSL features with causality into an SE model. The causal SSL features are encoded and combined with spectrogram features using feature-wise linear modulation to estimate a mask for enhancing the noisy input speech. Simultaneously, we quantize the causal SSL features using vector quantization to represent phonetic characteristics as semantic tokens. The model not only encodes SSL features but also predicts the future semantic tokens in multi-task learning (MTL). The experimental results using VoiceBank + DEMAND dataset show that our proposed method achieves 2.88 in PESQ, especially with semantic prediction MTL, in which we confirm that the semantic prediction played an important role in causal SE.