EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving

TL;DR

AR-MOT addresses tracking objects in driving videos based on audio cues, combining audio-visual understanding with robust object association. EchoTrack introduces a dual-stream transformer architecture with Bidirectional Frequency-domain Cross-Attention Fusion Module (Bi-FCFM) and Audio-visual Contrastive Tracking Learning (ACTL) to fuse frequency-aware audio cues with video features. The authors establish three large AR-MOT benchmarks—Echo-KITTI, Echo-KITTI+, and Echo-BDD—and demonstrate that EchoTrack achieves state-of-the-art results on both AR-MOT and RMOT tasks, highlighting the benefits of frequency-domain fusion and cross-modal contrastive learning for real-world autonomous driving. This work advances interactive, accessible, and safe driving systems by enabling natural audio-guided tracking and providing a comprehensive evaluation platform for future research.

Abstract

This paper introduces the task of Auditory Referring Multi-Object Tracking (AR-MOT), which dynamically tracks specific objects in a video sequence based on audio expressions and appears as a challenging problem in autonomous driving. Due to the lack of semantic modeling capacity in audio and video, existing works have mainly focused on text-based multi-object tracking, which often comes at the cost of tracking quality, interaction efficiency, and even the safety of assistance systems, limiting the application of such methods in autonomous driving. In this paper, we delve into the problem of AR-MOT from the perspective of audio-video fusion and audio-video tracking. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. The dual streams are intertwined with our Bidirectional Frequency-domain Cross-attention Fusion Module (Bi-FCFM), which bidirectionally fuses audio and video features from both frequency- and spatiotemporal domains. Moreover, we propose the Audio-visual Contrastive Tracking Learning (ACTL) regime to extract homogeneous semantic features between expressions and visual objects by learning homogeneous features between different audio and video objects effectively. Aside from the architectural design, we establish the first set of large-scale AR-MOT benchmarks, including Echo-KITTI, Echo-KITTI+, and Echo-BDD. Extensive experiments on the established benchmarks demonstrate the effectiveness of the proposed EchoTrack and its components. The source code and datasets are available at https://github.com/lab206/EchoTrack.

Figures (6)

• Figure 1: Overview of the introduced Auditory Referring Multi-Object Tracking (AR-MOT) task. The audio reference and the video are both fed into the model. The model is expected to track objects that are mentioned in the audio input step by step. Two samples from our Echo-KITTI+ dataset are provided to illustrate the workflow of the challenging AR-MOT.
• Figure 2: Comparison of tracking performance in HOTA luiten2021hota on the established Echo-BDD dataset with different conditions. MOTRv2 zhang2023motrv2 is a representative MOT method, and TransRMOT wu2023referring is a RMOT method. Both of them use the HuBERT-Base hsu2021hubert to encode the audio. The proposed EchoTrack consistently outperforms other methods across five conditions.
• Figure 3: Overview of the proposed EchoTrack. In a), EchoTrack comprises five primary components: from top to bottom, audio-video encoding, audio-visual feature fusion, audio-visual feature decoding, audio-visual tracking, and matching and loss optimization. In b), AGF stands for the adaptive gaussian filter, and Avg denotes the global average pooling operation.
• Figure 4: Overview of the AR-MOT benchmarks. The video includes $4$ attributes, i.e., Object, Scene, Weather, and Quality, our benchmarks encompasses $6$ objects, i.e., Pedestrian, Car, Motorcycle, Truck, Bus, Bicycle, $5$ scenarios, i.e., Road, City, Neighborhood, Business district, and Campus, $6$ weather conditions, i.e., Daylight, Night, Foggy, Snowy, Rainy, and Cloudy, $5$ video qualities, i.e., Normal, Blur, Overexposure, Underexposure, and Low-resolution. The audio expressions involve $4$ attributes of the objects, i.e., Position, Color, Movement, and Gender. It involves $4$ positions, i.e., Left, Right, Forward, and Opposite, $6$ colors, i.e., Red, Black, Blue, Gray, Yellow, and White, $5$ motions, i.e., Turning, Driving, Stopping, Walking, and Standing, $2$ genders, i.e., Male and Female.
• Figure 5: Qualitative results of different state-of-the-art methods include MOTRv2 zhang2023motrv2, TransRMOT wu2023referring, and the proposed EchoTrack on Echo-KITTI, Echo-KITTI+, and Echo-BDD datasets. EchoTrack shows leading tracking performance.