FabuLight-ASD: Unveiling Speech Activity via Body Language

TL;DR

FabuLight-ASD extends Light-ASD by incorporating body pose graphs to improve active speaker detection in challenging, real-world scenes. By fusing face, audio, and skeleton-based body features via a lightweight architecture, it achieves an overall mAP of 94.3% on the WASD dataset, outperforming the Light-ASD baseline while maintaining modest increases in parameters (27.3%) and MAC operations (up to 2.4%). The pose stream, based on ST-GCN, provides notable gains in scenarios with speech impairment, face occlusion, and heavy background noise, and the upper-body variant often offers the best trade-off between accuracy and efficiency. These results highlight the practical potential of pose-informed, multimodal ASD for robust, real-time human-robot interaction and related applications, with code and weights available publicly.

Abstract

Active speaker detection (ASD) in multimodal environments is crucial for various applications, from video conferencing to human-robot interaction. This paper introduces FabuLight-ASD, an advanced ASD model that integrates facial, audio, and body pose information to enhance detection accuracy and robustness. Our model builds upon the existing Light-ASD framework by incorporating human pose data, represented through skeleton graphs, which minimises computational overhead. Using the Wilder Active Speaker Detection (WASD) dataset, renowned for reliable face and body bounding box annotations, we demonstrate FabuLight-ASD's effectiveness in real-world scenarios. Achieving an overall mean average precision (mAP) of 94.3%, FabuLight-ASD outperforms Light-ASD, which has an overall mAP of 93.7% across various challenging scenarios. The incorporation of body pose information shows a particularly advantageous impact, with notable improvements in mAP observed in scenarios with speech impairment, face occlusion, and human voice background noise. Furthermore, efficiency analysis indicates only a modest increase in parameter count (27.3%) and multiply-accumulate operations (up to 2.4%), underscoring the model's efficiency and feasibility. These findings validate the efficacy of FabuLight-ASD in enhancing ASD performance through the integration of body pose data. FabuLight-ASD's code and model weights are available at https://github.com/knowledgetechnologyuhh/FabuLight-ASD.

Figures (8)

• Figure 1: The architecture of FabuLight-ASD. The model determines whether a target individual is actively speaking or silent in each video frame based on the face crops and body poses of that individual across all frames, along with the corresponding audio information. The overall architecture builds on Light-ASD, with the body feature encoder being a newly added component that improves performance.
• Figure 2: Examples of WASD videos of various categories.
• Figure 3: Examples of bounding box inaccuracies in the AVA-ActiveSpeaker dataset. Correctly placed bounding boxes are indicated in blue and mistake cases in red bounding boxes.
• Figure 4: Architectures of Light-ASD feature encoders and their inner blocks. Blocks are given by their number of in-channels $C_{in}$ and out-channels $C_{out}$, and in the face feature encoder, the stride $s$ in both spatial dimensions. Pooling layer parameters are the kernel size, stride, and padding size. In the face feature encoder, pooling is applied on both spatial dimensions, while in the audio feature encoder, it is applied on the temporal dimension. The parameters of the convolution layers within the blocks are the number of in-channels, number of out-channels, kernel size, stride, and padding size. The labels beside each convolution layer indicate whether the convolution is applied on the spatial or temporal dimensions.
• Figure 5: FabuLight-ASD's body encoder. The parameters of each body block indicate the number of in- and out-channels.