Efficient Face Detection with Audio-Based Region Proposals for Human-Robot Interactions

TL;DR

This work tackles the computational bottleneck of real-time face detection on robots by introducing audio-driven ROI proposals. It integrates a VAD gate, a denoising stage, and a steered-beamformer SSL to generate acoustic images that guide a ROI selector, reducing pixel processing to regions around the active speaker. The system, implemented with a low-cost acoustic camera and YuNet for face detection, achieves substantial speedups (up to $1.95\times$ runtime reduction and $3.11\times$ FLOPs reduction at high SNR) while maintaining reasonable face recall (over 90% at high SNR, ~71% at $0$ dB). This approach enables context-aware, efficient perception for human-robot interaction and is adaptable to other applications such as surveillance or video conferencing.

Abstract

Efficient face detection is critical to provide natural human-robot interactions. However, computer vision tends to involve a large computational load due to the amount of data (i.e. pixels) that needs to be processed in a short amount of time. This is undesirable on robotics platforms where multiple processes need to run in parallel and where the processing power is limited by portability constraints. Existing solutions often involve reducing image quality which can negatively impact processing. The literature also reports methods to generate regions of interest in images from pixel data. Although it is a promising idea, these methods often involve heavy vision algorithms. In this paper, we evaluate how audio can be used to generate regions of interest in optical images to reduce the number of pixels to process with computer vision. Thereby, we propose a unique attention mechanism to localize a speech source and evaluate its impact on an existing face detection algorithm. Our results show that the attention mechanism reduces the computational load and offers an interesting trade-off between speed and accuracy. The proposed pipeline is flexible and can be easily adapted to other applications such as robot surveillance, video conferences or smart glasses.

Figures (9)

• Figure 1: Overview of the proposed pipeline.
• Figure 2: Acoustic image example with with resolutions of (a) $640 \times 480$ regions and (b) $9 \times 7$ regions.
• Figure 3: Acoustic camera developed for the project.
• Figure 4: Architecture for the VAD module (a) and the denoising module (b). The VAD network has 61,703 parameters while the denoising network has 379,907 parameters.
• Figure 5: Data augmentation to generate training samples for VAD and denoising modules -- 10% voice only, 15% noise only, 40% voice + FSD50K noise, 30% voice + MusicNet noise and 5% voice + white noise.