Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Lips: Integrating Gesture and Lip Cues for Robust Audio-visual Speaker Extraction

Zexu Pan, Xinyuan Qian, Shengkui Zhao, Kun Zhou, Bin Ma

TL;DR

Addressing the cocktail party problem, the paper tackles robust audio-visual speaker extraction by exploiting co-speech upper-body gestures in addition to lip movements. It introduces SeLG, which uses cross-attention to fuse lip and gesture cues querying the audio mixture, and adds a contrastive InfoNCE loss to align gesture embeddings with lip dynamics. Experiments on the YGD-2mix and YGD-3mix datasets show improved performance and robustness under missing modalities compared to unimodal and concatenation baselines. This work expands AVSE by leveraging co-speech gestures, enabling robust extraction in real-world settings with occlusions or distant views.

Abstract

Most audio-visual speaker extraction methods rely on synchronized lip recording to isolate the speech of a target speaker from a multi-talker mixture. However, in natural human communication, co-speech gestures are also temporally aligned with speech, often emphasizing specific words or syllables. These gestures provide complementary visual cues that can be especially valuable when facial or lip regions are occluded or distant. In this work, we move beyond lip-centric approaches and propose SeLG, a model that integrates both lip and upper-body gesture information for robust speaker extraction. SeLG features a cross-attention-based fusion mechanism that enables each visual modality to query and selectively attend to relevant speech features in the mixture. To improve the alignment of gesture representations with speech dynamics, SeLG also employs a contrastive InfoNCE loss that encourages gesture embeddings to align more closely with corresponding lip embeddings, which are more strongly correlated with speech. Experimental results on the YGD dataset, containing TED talks, demonstrate that the proposed contrastive learning strategy significantly improves gesture-based speaker extraction, and that our proposed SeLG model, by effectively fusing lip and gesture cues with an attention mechanism and InfoNCE loss, achieves superior performance compared to baselines, across both complete and partial (i.e., missing-modality) conditions.

Beyond Lips: Integrating Gesture and Lip Cues for Robust Audio-visual Speaker Extraction

TL;DR

Addressing the cocktail party problem, the paper tackles robust audio-visual speaker extraction by exploiting co-speech upper-body gestures in addition to lip movements. It introduces SeLG, which uses cross-attention to fuse lip and gesture cues querying the audio mixture, and adds a contrastive InfoNCE loss to align gesture embeddings with lip dynamics. Experiments on the YGD-2mix and YGD-3mix datasets show improved performance and robustness under missing modalities compared to unimodal and concatenation baselines. This work expands AVSE by leveraging co-speech gestures, enabling robust extraction in real-world settings with occlusions or distant views.

Abstract

Most audio-visual speaker extraction methods rely on synchronized lip recording to isolate the speech of a target speaker from a multi-talker mixture. However, in natural human communication, co-speech gestures are also temporally aligned with speech, often emphasizing specific words or syllables. These gestures provide complementary visual cues that can be especially valuable when facial or lip regions are occluded or distant. In this work, we move beyond lip-centric approaches and propose SeLG, a model that integrates both lip and upper-body gesture information for robust speaker extraction. SeLG features a cross-attention-based fusion mechanism that enables each visual modality to query and selectively attend to relevant speech features in the mixture. To improve the alignment of gesture representations with speech dynamics, SeLG also employs a contrastive InfoNCE loss that encourages gesture embeddings to align more closely with corresponding lip embeddings, which are more strongly correlated with speech. Experimental results on the YGD dataset, containing TED talks, demonstrate that the proposed contrastive learning strategy significantly improves gesture-based speaker extraction, and that our proposed SeLG model, by effectively fusing lip and gesture cues with an attention mechanism and InfoNCE loss, achieves superior performance compared to baselines, across both complete and partial (i.e., missing-modality) conditions.
Paper Structure (19 sections, 2 equations, 3 figures, 1 table)

This paper contains 19 sections, 2 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Model architecture
  4. Gesture encoder
  5. Lip encoder
  6. Speech encoder and decoder
  7. Separator
  8. Objective function
  9. Experimental setup
  10. Dataset
  11. Baselines
  12. Model configuration
  13. Training
  14. Results
  15. Effects of using both lip and gesture cues
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Our proposed audio-visual speaker extraction model named SeLG, which estimates the target speech conditioned on the synchronized gesture sequence $v_g(t)$ and the lip sequence $v_l(t)$ of the target speaker. The symbols $\oplus$ and $\otimes$ represent element-wise addition and multiplication, respectively.
  • Figure 2: The SI-SNRi histogram of YGD-2mix test samples.
  • Figure 3: The SI-SNRi histogram of YGD-3mix test samples.