Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directions

Jinzheng Zhao, Yong Xu, Xinyuan Qian, Davide Berghi, Peipei Wu, Meng Cui, Jianyuan Sun, Philip J. B. Jackson, Wenwu Wang

TL;DR

Audio-visual speaker tracking leverages complementary cues from sound and vision to localize and follow speakers in dynamic environments. The survey maps measurement extraction, Bayesian trackers, variational and differentiable Bayesian methods, and deep-learning-based tracking, linking audio-visual fusion, 2D-3D position conversion, and data association to established datasets (AV16.3, CAV3D, AVRI) and metrics. It highlights the strengths and limitations of parametric and learning-based approaches, notes the emergent role of differentiable Bayesian filters, and discusses practical challenges such as data requirements and real-time constraints. The paper outlines future directions including speech separation integration, distributed sensing, egocentric contexts, and prompt-based tracking to advance robust, scalable multi-speaker tracking in real-world settings.

Abstract

Audio-visual speaker tracking has drawn increasing attention over the past few years due to its academic values and wide applications. Audio and visual modalities can provide complementary information for localization and tracking. With audio and visual information, the Bayesian-based filter and deep learning-based methods can solve the problem of data association, audio-visual fusion and track management. In this paper, we conduct a comprehensive overview of audio-visual speaker tracking. To our knowledge, this is the first extensive survey over the past five years. We introduce the family of Bayesian filters and summarize the methods for obtaining audio-visual measurements. In addition, the existing trackers and their performance on the AV16.3 dataset are summarized. In the past few years, deep learning techniques have thrived, which also boost the development of audio-visual speaker tracking. The influence of deep learning techniques in terms of measurement extraction and state estimation is also discussed. Finally, we discuss the connections between audio-visual speaker tracking and other areas such as speech separation and distributed speaker tracking.

Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directions

TL;DR

Audio-visual speaker tracking leverages complementary cues from sound and vision to localize and follow speakers in dynamic environments. The survey maps measurement extraction, Bayesian trackers, variational and differentiable Bayesian methods, and deep-learning-based tracking, linking audio-visual fusion, 2D-3D position conversion, and data association to established datasets (AV16.3, CAV3D, AVRI) and metrics. It highlights the strengths and limitations of parametric and learning-based approaches, notes the emergent role of differentiable Bayesian filters, and discusses practical challenges such as data requirements and real-time constraints. The paper outlines future directions including speech separation integration, distributed sensing, egocentric contexts, and prompt-based tracking to advance robust, scalable multi-speaker tracking in real-world settings.

Abstract

Audio-visual speaker tracking has drawn increasing attention over the past few years due to its academic values and wide applications. Audio and visual modalities can provide complementary information for localization and tracking. With audio and visual information, the Bayesian-based filter and deep learning-based methods can solve the problem of data association, audio-visual fusion and track management. In this paper, we conduct a comprehensive overview of audio-visual speaker tracking. To our knowledge, this is the first extensive survey over the past five years. We introduce the family of Bayesian filters and summarize the methods for obtaining audio-visual measurements. In addition, the existing trackers and their performance on the AV16.3 dataset are summarized. In the past few years, deep learning techniques have thrived, which also boost the development of audio-visual speaker tracking. The influence of deep learning techniques in terms of measurement extraction and state estimation is also discussed. Finally, we discuss the connections between audio-visual speaker tracking and other areas such as speech separation and distributed speaker tracking.
Paper Structure (54 sections, 21 equations, 5 figures, 10 tables)

This paper contains 54 sections, 21 equations, 5 figures, 10 tables.

Table of Contents

  1. INTRODUCTION
  2. Measurements
  3. Visual Measurements
  4. Parametric-Based Methods
  5. Learning-Based methods
  6. Comparison between Parametric-Based Methods and Learning-Based Methods
  7. Audio Measurements
  8. Parametric-Based Methods
  9. Global Coherence Field
  10. GCC-PHAT de-emphasis
  11. stGCF
  12. Learning-Based Methods
  13. Comparison between Parametric-Based Methods and Learning-Based Methods
  14. Extracting Features in Teacher-Student Paradigm
  15. Methods of Audio-Visual Speaker Tracking
  16. ...and 39 more sections

Figures (5)

  • Figure 1: The structure of audio-visual speaker tracker.
  • Figure 2: The Classification of Audio-Visual Trackers.
  • Figure 3: Illustrations of Hidden Markov Model
  • Figure 4: States Updating of Bayesian Filter
  • Figure 5: The Camera Projection Model