Dual-View Predictive Diffusion: Lightweight Speech Enhancement via Spectrogram-Image Synergy

TL;DR

DVPD tackles the computational burden of diffusion-based speech enhancement by exploiting the dual nature of spectrograms as both acoustic representations and visual textures. It introduces a Frequency-Adaptive Non-uniform Compression (FANC) encoder and a Lightweight Image-based Spectro-Awareness (LISA) module to reduce spectral waste, paired with a Training-free Lossless Boost (TLB) for inference-time refinement without retraining. The model adopts a parallel predictive-diffusion architecture that achieves state-of-the-art quality with only 35% of the parameters and 40% of the MACs of the previous SOTA lightweight baseline, PGUSE, while maintaining strong universal robustness across distortions. Extensive experiments on WSJ0-UNI USE and related benchmarks validate both efficiency and performance, underscoring the practical impact of the dual-view design for real-world speech enhancement.

Abstract

Diffusion models have recently set new benchmarks in Speech Enhancement (SE). However, most existing score-based models treat speech spectrograms merely as generic 2D images, applying uniform processing that ignores the intrinsic structural sparsity of audio, which results in inefficient spectral representation and prohibitive computational complexity. To bridge this gap, we propose DVPD, an extremely lightweight Dual-View Predictive Diffusion model, which uniquely exploits the dual nature of spectrograms as both visual textures and physical frequency-domain representations across both training and inference stages. Specifically, during training, we optimize spectral utilization via the Frequency-Adaptive Non-uniform Compression (FANC) encoder, which preserves critical low-frequency harmonics while pruning high-frequency redundancies. Simultaneously, we introduce a Lightweight Image-based Spectro-Awareness (LISA) module to capture features from a visual perspective with minimal overhead. During inference, we propose a Training-free Lossless Boost (TLB) strategy that leverages the same dual-view priors to refine generation quality without any additional fine-tuning. Extensive experiments across various benchmarks demonstrate that DVPD achieves state-of-the-art performance while requiring only 35% of the parameters and 40% of the inference MACs compared to SOTA lightweight model, PGUSE. These results highlight DVPD's superior ability to balance high-fidelity speech quality with extreme architectural efficiency. Code and audio samples are available at the anonymous website: {https://anonymous.4open.science/r/dvpd_demo-E630}

Figures (10)

• Figure 1: Dual-view of spectrogram frequency bands. Left: Acoustic perspective; Middle and Right: Visual image perspectives of low-frequency and high-frequency components, respectively.
• Figure 2: Architectural overview of the proposed DVPD. (A) The overall symmetrical dual-branch structure of DVPD, comprising the predictive branch (upper) and the diffusion branch (lower). (B) Detailed design of the Frequency-Adaptive Non-uniform Compression (FANC) Encoder. (C) Detailed design of enhancement network. (D) Illustration of the Frequency-Aware Interaction module.
• Figure 3: Detailed architecture of the LISA module.
• Figure 4: Operations of the TLB strategy. Scaling factors $b$ and $s$ modulate the intensity of backbone and skip features, respectively.
• Figure 5: Performance comparison across diverse benchmarks. All models were trained exclusively on the WSJ0-UNI to evaluate their universal generalization capabilities. Dashed lines represent predictive models, while solid lines denote diffusion-based generative models.