Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-Time Speech Enhancement via a Hybrid ViT: A Dual-Input Acoustic-Image Feature Fusion

Behnaz Bahmei, Siamak Arzanpour, Elina Birmingham

TL;DR

This work tackles real-time, single-channel speech enhancement in environments with non-stationary noise by introducing a lightweight hybrid Vision Transformer that fuses spectrogram-based acoustic-image features with raw audio cues. The model outputs two ratio masks for clean and noise components via a dual-input architecture with spectral and temporal branches, enabling effective denoising while maintaining low latency suitable for embedded devices (frame length $16$ ms and end-to-end latency $<35$ ms). Key contributions include an adapted ViT for 2D acoustic inputs, Gaussian-smoothed dual masks, and a practical real-time pipeline validated on LibriSpeech with UrbanSound8K and Google AudioSet, showing substantial improvements in PESQ, STOI, Seg SNR, and LLR across dynamic noises. The findings demonstrate robust performance in realistic conditions and highlight the approach's suitability for edge deployments in real-time speech applications. In short, the paper delivers a practical, transformer-based, dual-input solution that preserves speech quality and intelligibility in challenging, real-world noise scenarios.

Abstract

Speech quality and intelligibility are significantly degraded in noisy environments. This paper presents a novel transformer-based learning framework to address the single-channel noise suppression problem for real-time applications. Although existing deep learning networks have shown remarkable improvements in handling stationary noise, their performance often diminishes in real-world environments characterized by non-stationary noise (e.g., dog barking, baby crying). The proposed dual-input acoustic-image feature fusion using a hybrid ViT framework effectively models both temporal and spectral dependencies in noisy signals. Designed for real-world audio environments, the proposed framework is computationally lightweight and suitable for implementation on embedded devices. To evaluate its effectiveness, four standard and commonly used quality measurements, namely PESQ, STOI, Seg SNR, and LLR, are utilized. Experimental results obtained using the Librispeech dataset as the clean speech source and the UrbanSound8K and Google Audioset datasets as the noise sources, demonstrate that the proposed method significantly improves noise reduction, speech intelligibility, and perceptual quality compared to the noisy input signal, achieving performance close to the clean reference.

Real-Time Speech Enhancement via a Hybrid ViT: A Dual-Input Acoustic-Image Feature Fusion

TL;DR

This work tackles real-time, single-channel speech enhancement in environments with non-stationary noise by introducing a lightweight hybrid Vision Transformer that fuses spectrogram-based acoustic-image features with raw audio cues. The model outputs two ratio masks for clean and noise components via a dual-input architecture with spectral and temporal branches, enabling effective denoising while maintaining low latency suitable for embedded devices (frame length ms and end-to-end latency ms). Key contributions include an adapted ViT for 2D acoustic inputs, Gaussian-smoothed dual masks, and a practical real-time pipeline validated on LibriSpeech with UrbanSound8K and Google AudioSet, showing substantial improvements in PESQ, STOI, Seg SNR, and LLR across dynamic noises. The findings demonstrate robust performance in realistic conditions and highlight the approach's suitability for edge deployments in real-time speech applications. In short, the paper delivers a practical, transformer-based, dual-input solution that preserves speech quality and intelligibility in challenging, real-world noise scenarios.

Abstract

Speech quality and intelligibility are significantly degraded in noisy environments. This paper presents a novel transformer-based learning framework to address the single-channel noise suppression problem for real-time applications. Although existing deep learning networks have shown remarkable improvements in handling stationary noise, their performance often diminishes in real-world environments characterized by non-stationary noise (e.g., dog barking, baby crying). The proposed dual-input acoustic-image feature fusion using a hybrid ViT framework effectively models both temporal and spectral dependencies in noisy signals. Designed for real-world audio environments, the proposed framework is computationally lightweight and suitable for implementation on embedded devices. To evaluate its effectiveness, four standard and commonly used quality measurements, namely PESQ, STOI, Seg SNR, and LLR, are utilized. Experimental results obtained using the Librispeech dataset as the clean speech source and the UrbanSound8K and Google Audioset datasets as the noise sources, demonstrate that the proposed method significantly improves noise reduction, speech intelligibility, and perceptual quality compared to the noisy input signal, achieving performance close to the clean reference.

Paper Structure

This paper contains 10 sections, 10 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Pre-processing
  4. Model Structure
  5. Dual Mask Prediction
  6. Post-processing
  7. Experimental Setup
  8. Real-time Processing
  9. Results and Discussion
  10. Conclusion

Figures (4)

  • Figure 1: The proposed noise suppression architecture.
  • Figure 2: The model structure.
  • Figure 3: Spectrogram performance of the proposed method for different noise structure types.
  • Figure 4: A comparison between the estimated masks by the proposed method and the real masks.