Robust Audio-Visual Target Speaker Extraction with Emotion-Aware Multiple Enrollment Fusion

TL;DR

It is demonstrated that fusing the complementary one frame of face image with frame-level lip features achieves both strong performance and robustness for the AVTSE task, and training with a high missing rate dramatically enhances robustness.

Abstract

Audio-Visual Target Speaker Extraction (AVTSE) is crucial for cocktail party scenarios. Leveraging multiple cues --such as utterance-level speaker embeddings or steady face images, and frame-level lip motion or facial expression features --can significantly improve performance. However, real-world applications often suffer from intermittent signal loss, especially for frame-level cues. This paper systematically investigates the robustness of multi-enrollment fusion under varying degrees of modality missing. Results show that while full multimodal fusion excels under ideal conditions, its performance degrades sharply when encountering unseen modalities missing during the testing. Crucially, training with a high missing rate dramatically enhances robustness, maintaining stable performance even under severe test-time modality missing. We demonstrate that fusing the complementary one frame of face image with frame-level lip features achieves both strong performance and robustness for the AVTSE task. The model and codes are shared.

Figures (1)

• Figure 1: Proposed AVTSE system structure with multiple enrollment fusion