Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attention

TL;DR

Experiments showed that Dolphin not only surpassed the current state-of-the-art (SOTA) model in separation quality but also achieved remarkable improvements in efficiency: over 50% fewer parameters, more than 2.4x reduction in MACs, and over 6x faster GPU inference speed.

Abstract

Audio-visual speech separation (AVSS) methods leverage visual cues to extract target speech and have demonstrated strong separation quality in noisy acoustic environments. However, these methods usually involve a large number of parameters and require high computational cost, which is unacceptable in many applications where speech separation serves as only a preprocessing step for further speech processing. To address this issue, we propose an efficient AVSS method, named Dolphin. For visual feature extraction, we develop DP-LipCoder, a dual-path lightweight video encoder that transforms lip-motion into discrete audio-aligned semantic tokens. For audio separation, we construct a lightweight encoder-decoder separator, in which each layer incorporates a global-local attention (GLA) block to efficiently capture multi-scale dependencies. Experiments on three benchmark datasets showed that Dolphin not only surpassed the current state-of-the-art (SOTA) model in separation quality but also achieved remarkable improvements in efficiency: over 50% fewer parameters, more than 2.4x reduction in MACs, and over 6x faster GPU inference speed. These results indicate that Dolphin offers a practical and deployable solution for high-performance AVSS in real-world scenarios. Our code and demo page are publicly available at http://cslikai.cn/Dolphin/.

Figures (9)

• Figure 1: The overall pipeline of Dolphin. Frozen parameters are displayed with a snowflake marker.
• Figure 2: Overall pipeline of the AVDP-MagVIT network. The stacked 3D residual blocks denote two 3D residual blocks connected sequentially. The three loss functions are shown as gray blocks.
• Figure 3: The architecture of separator. Here, we set the encoder and decoder layers $Q$ to 4.
• Figure 4: Detailed architectures of GLA blocks in the separator.
• Figure 5: Comparison of heat diffusion filtering (red dashed, $\alpha = 0.8, t = 1.2$) and Gaussian convolution (orange solid, K = 21) applied to a test signal containing multiple frequency components and local impulses (blue dotted). The inset shows a magnified view of the impulse response region (green dashed rectangle), revealing superior edge preservation by the heat diffusion method while maintaining effective noise suppression.