ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection

TL;DR

ASDnB tackles robustness in Active Speaker Detection by fusing facial cues with body dynamics within a single visual encoder. It employs a 2D+1D convolutional design to reduce computation, adaptive weight scheduling to balance visual and audio-visual cues, and Bidirectional GRU temporal modeling with MFCC-based audio encoding. The approach yields state-of-the-art results on AVA-ActiveSpeaker, WASD, and cross-domain Columbia, supported by extensive ablations. This lightweight, intra-encoder fusion strategy enhances ASD performance in wild conditions with occlusions or degraded inputs, offering a strong practical baseline.

Abstract

State-of-the-art Active Speaker Detection (ASD) approaches mainly use audio and facial features as input. However, the main hypothesis in this paper is that body dynamics is also highly correlated to "speaking" (and "listening") actions and should be particularly useful in wild conditions (e.g., surveillance settings), where face cannot be reliably accessed. We propose ASDnB, a model that singularly integrates face with body information by merging the inputs at different steps of feature extraction. Our approach splits 3D convolution into 2D and 1D to reduce computation cost without loss of performance, and is trained with adaptive weight feature importance for improved complement of face with body data. Our experiments show that ASDnB achieves state-of-the-art results in the benchmark dataset (AVA-ActiveSpeaker), in the challenging data of WASD, and in cross-domain settings using Columbia. This way, ASDnB can perform in multiple settings, which is positively regarded as a strong baseline for robust ASD models (code available at https://github.com/Tiago-Roxo/ASDnB).

Figures (7)

• Figure 1: State-of-the-art ASD models solely rely on facial cues as visual input to perform. This approach is only reliable with cooperative (subjects) conditions, increasing uncertainty in model prediction in more challenging settings. Our approach aims to complement face with body cues to create more robust models that are able to perform in cooperative and unconstrained scenarios.
• Figure 2: Overview of the ASDnB architecture. Body and face data are fed into the visual encoder, allowing intra encoder feature fusion and complement, while audio is processed through its respective encoder. Audio and visual features are combined to predict subject speaking label, using an adaptive weighted loss for combined and visual features ($\mathcal{L}_{av}$ and $\mathcal{L}_{v}$, respectively).
• Figure 3: Overview of the flow of face and body combination in ASDnB visual encoder. The first convolution for both inputs downsamples via stride.
• Figure 4: Bidirectional GRUs of visual and audio combination in ASDnB classifier, before inputting to FC layers for speaker classification.
• Figure 5: Comparison of ASDnB performance relative to ASD state-of-the-art models for (a) number of faces per frame and (b) various face sizes in AVA-ActiveSpeaker.