Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-Time Object Tracking with On-Device Deep Learning for Adaptive Beamforming in Dynamic Acoustic Environments

Jorge Ortigoso-Narro, Jose A. Belloch, Adrian Amor-Martin, Sandra Roger, Maximo Cobos

TL;DR

The paper addresses real-time sound source localization and directional audio capture in dynamic environments by fusing on-device deep learning–based vision with adaptive beamforming. It proposes an integrated pipeline consisting of a vision module with depth estimation, a planar concentric MEMS microphone array for 2D beam steering, and an NVIDIA Jetson Orin Nano for closed-loop perception and steering. A frequency-domain delay-and-sum beamformer guided by DoA estimates, combined with a parallel, timestamp-synchronized multimodal processing chain and an overlap-add reconstruction, achieves end-to-end latency well under 100 ms. Experimental results in anechoic and dynamic room setups show consistent SIR improvements and robust source tracking, highlighting the approach’s potential for teleconferencing, smart home devices, and assistive technologies.

Abstract

Advances in object tracking and acoustic beamforming are driving new capabilities in surveillance, human-computer interaction, and robotics. This work presents an embedded system that integrates deep learning-based tracking with beamforming to achieve precise sound source localization and directional audio capture in dynamic environments. The approach combines single-camera depth estimation and stereo vision to enable accurate 3D localization of moving objects. A planar concentric circular microphone array constructed with MEMS microphones provides a compact, energy-efficient platform supporting 2D beam steering across azimuth and elevation. Real-time tracking outputs continuously adapt the array's focus, synchronizing the acoustic response with the target's position. By uniting learned spatial awareness with dynamic steering, the system maintains robust performance in the presence of multiple or moving sources. Experimental evaluation demonstrates significant gains in signal-to-interference ratio, making the design well-suited for teleconferencing, smart home devices, and assistive technologies.

Real-Time Object Tracking with On-Device Deep Learning for Adaptive Beamforming in Dynamic Acoustic Environments

TL;DR

The paper addresses real-time sound source localization and directional audio capture in dynamic environments by fusing on-device deep learning–based vision with adaptive beamforming. It proposes an integrated pipeline consisting of a vision module with depth estimation, a planar concentric MEMS microphone array for 2D beam steering, and an NVIDIA Jetson Orin Nano for closed-loop perception and steering. A frequency-domain delay-and-sum beamformer guided by DoA estimates, combined with a parallel, timestamp-synchronized multimodal processing chain and an overlap-add reconstruction, achieves end-to-end latency well under 100 ms. Experimental results in anechoic and dynamic room setups show consistent SIR improvements and robust source tracking, highlighting the approach’s potential for teleconferencing, smart home devices, and assistive technologies.

Abstract

Advances in object tracking and acoustic beamforming are driving new capabilities in surveillance, human-computer interaction, and robotics. This work presents an embedded system that integrates deep learning-based tracking with beamforming to achieve precise sound source localization and directional audio capture in dynamic environments. The approach combines single-camera depth estimation and stereo vision to enable accurate 3D localization of moving objects. A planar concentric circular microphone array constructed with MEMS microphones provides a compact, energy-efficient platform supporting 2D beam steering across azimuth and elevation. Real-time tracking outputs continuously adapt the array's focus, synchronizing the acoustic response with the target's position. By uniting learned spatial awareness with dynamic steering, the system maintains robust performance in the presence of multiple or moving sources. Experimental evaluation demonstrates significant gains in signal-to-interference ratio, making the design well-suited for teleconferencing, smart home devices, and assistive technologies.

Paper Structure

This paper contains 20 sections, 13 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed system
  3. Vision Module
  4. Object classification
  5. Depth Estimation
  6. Source Tracking
  7. Beamforming Module
  8. Frequency-Domain Delay-and-Sum Beamformer
  9. Steering delay computation.
  10. Frequency-domain formulation.
  11. Hardware and Microphone Array Desing
  12. Optimizing for Real-Time Operation
  13. Parallel Multimodal Processing
  14. GPU-Accelerated Perception and Frequency-Domain Beamforming
  15. Overlap–Add Beamforming Implementation
  16. ...and 5 more sections

Figures (10)

  • Figure 1: High-level block diagram design.
  • Figure 2: System geometry for DoA estimation.
  • Figure 3: Designed PCB containing the MEMS microphones. The sensors are placed along $R=3$ rings of radii $\rho = \{0, 2.5, 4.5\}\,$cm and equally spaced along each ring: for $r=2$, $\Delta\phi = \pi/2\,$ rad and for $r=3$, $\Delta\phi = \pi/4\,$ rad.
  • Figure 4: Implemented thread diagram and acquisition/processing pipeline. For acronym definition see Sec. \ref{['subsec:overlapadd']}.
  • Figure 5: Arbitrary CUDA frequency-domain beamforming annotated range profiling with NVTX and Nsight Systems. A 10-second warm-up period was performed prior to measurement to ensure that initial memory and cuFFT allocations did not affect timing results.
  • ...and 5 more figures