MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues

TL;DR

MoMuSE tackles the challenge of maintaining target speaker extraction in audio-visual mixtures when visual cues are impaired or missing. It extends MuSE with a momentum-based memory bank of anchor embeddings and an Anchor Speaker Embedding Updating (ASEU) module, enabling dynamic fusion of current and historical voiceprints for robust, real-time extraction. The method is trained with Segment-Level Optimization, parameter initialization from a MuSE checkpoint, and utterance-level optimization, and it uses a momentum mechanism that initializes from the first window and updates embeddings when reliable cues are detected. Empirical results show that MoMuSE significantly improves performance under visual impairments, particularly in total visual absence, demonstrating the practical value of momentum-based memory in AV-TSE without audio pre-enrollment.

Abstract

Audio-visual Target Speaker Extraction (AV-TSE) aims to isolate the speech of a specific target speaker from an audio mixture using time-synchronized visual cues. In real-world scenarios, visual cues are not always available due to various impairments, which undermines the stability of AV-TSE. Despite this challenge, humans can maintain attentional momentum over time, even when the target speaker is not visible. In this paper, we introduce the Momentum Multi-modal target Speaker Extraction (MoMuSE), which retains a speaker identity momentum in memory, enabling the model to continuously track the target speaker. Designed for real-time inference, MoMuSE extracts the current speech window with guidance from both visual cues and dynamically updated speaker momentum. Experimental results demonstrate that MoMuSE exhibits significant improvement, particularly in scenarios with severe impairment of visual cues.

Figures (3)

• Figure 1: Examples of normal and impaired visual frames.
• Figure 2: (a) Overall structure of MoMuSE. Modules in gray denote the original structures from MuSE, while modules in other colors denote the proposed new structures. (b) Speaker Extractor. (c) The detailed structure of Anchor Speaker Embedding Updating (ASEU), which is an attention based module. $\otimes$, $\ominus$ and $\oplus$ refer to point-wise multiplication, concatenation and point-wise addition.
• Figure 3: Model performance under varied impaired ratios is shown, where MoMuSE_w/o(PI+$\mathcal{L}_{\text{Pe}}$) excludes parameter initialization (PI) and penalty loss ($\mathcal{L}_{\text{Pe}}$), and MoMuSE_w/o($\mathcal{L}_{\text{Pe}}$) excludes only $\mathcal{L}_{\text{Pe}}$