LASER: Lip Landmark Assisted Speaker Detection for Robustness

TL;DR

This work addresses Active Speaker Detection (ASD) under challenging real-world conditions where lip-audio synchronization can be imperfect. It introduces LASER, which guides visual representations by encoding 2D lip landmark tracks into dense feature maps and fusing them with frame-level visual features; an auxiliary consistency loss ensures robustness when lip landmarks are unavailable at test time. The approach retains compatibility with existing ASD architectures and demonstrates substantial robustness gains, particularly in noisy environments, supported by LASER-bench, a curated noise-rich dataset. Empirically, LASER outperforms state-of-the-art methods on standard benchmarks and shows pronounced improvements under high background noise and audio-visual misalignment, underscoring its practical impact for real-world audiovisual systems.

Abstract

Active Speaker Detection (ASD) aims to identify who is speaking in complex visual scenes. While humans naturally rely on lip-audio synchronization, existing ASD models often misclassify non-speaking instances when lip movements and audio are unsynchronized. To address this, we propose Lip landmark Assisted Speaker dEtection for Robustness (LASER), which explicitly incorporates lip landmarks during training to guide the model's attention to speech-relevant regions. Given a face track, LASER extracts visual features and encodes 2D lip landmarks into dense maps. To handle failure cases such as low resolution or occlusion, we introduce an auxiliary consistency loss that aligns lip-aware and face-only predictions, removing the need for landmark detectors at test time. LASER outperforms state-of-the-art models across both in-domain and out-of-domain benchmarks. To further evaluate robustness in realistic conditions, we introduce LASER-bench, a curated dataset of modern video clips with varying levels of background noise. On the high-noise subset, LASER improves mAP by 3.3 and 4.3 points over LoCoNet and TalkNet, respectively, demonstrating strong resilience to real-world acoustic challenges.

Figures (7)

• Figure 1: Qualitative comparison of Grad-Cam grad-cam visualizations for LoCoNet and our LASER under nonsynchronized audiovisual scenarios. LoCoNet wang2024loconet struggles to accurately predict "not speaking" when visual frames and audio tracks are misaligned, often failing to focus on the lip region when making these predictions. In contrast, our LASER consistently concentrates on the lip area and successfully identifies "not speaking" situations.
• Figure 2: Illustration of LASER. Given a face track $V$, we first obtain a lip landmark track using a facial landmark detector and encode the 2D coordinates of these landmarks into continuous 2D feature maps. These maps are then aggregated through a 1x1 convolution layer. The encoded lip track is concatenated with visual features from a 3D CNN and fed into ResNet and V-TCN to capture a temporal visual representation, which is further processed by context modeling modules wang2024loconettao2021someoneliao2023light to produce the final prediction. The consistency loss is computed between predictions made with and without lip landmark encoding and gradients are only propagated through the prediction without lip landmark. This way, the model is robust against missing lip landmarks at test time.
• Figure 3: Evaluation with unsynchronized audios. We use the same datasets as the evaluation of synchronized audios and report the per-frame accuracy on in-domain datasets (AVA-ActiveSpeaker) and out-of-domain datasets (Talkies and ASW). LASER consistently outperforms LoCoNet wang2024loconet under this evaluation protocol.
• Figure 4: Qualitative Comparison on LASER-bench. a) Ground-truth annotation. b) Detection results from our method LASER. c) Detection Results from baseline LoCoNet. Red: not speaking; Green: speaking; orange: incorrect predictions.
• Figure A: Our implementation of Landmark Pooling Wang2019landmark_pooling.