Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependencies

TL;DR

This work introduces an end-to-end joint audio-visual IVD task designed to detect vehicles visually under three states: moving, idling and engine off, and proposes AVIVD-Net, a novel network that integrates audio and visual features through a bidirectional attention mechanism.

Abstract

Idling vehicle detection (IVD) can be helpful in monitoring and reducing unnecessary idling and can be integrated into real-time systems to address the resulting pollution and harmful products. The previous approach [13], a non-end-to-end model, requires extra user clicks to specify a part of the input, making system deployment more error-prone or even not feasible. In contrast, we introduce an end-to-end joint audio-visual IVD task designed to detect vehicles visually under three states: moving, idling and engine off. Unlike feature co-occurrence task such as audio-visual vehicle tracking, our IVD task addresses complementary features, where labels cannot be determined by a single modality alone. To this end, we propose AVIVD-Net, a novel network that integrates audio and visual features through a bidirectional attention mechanism. AVIVD-Net streamlines the input process by learning a joint feature space, reducing the deployment complexity of previous methods. Additionally, we introduce the AVIVD dataset, which is seven times larger than previous datasets, offering significantly more annotated samples to study the IVD problem. Our model achieves performance comparable to prior approaches, making it suitable for automated deployment. Furthermore, by evaluating AVIVDNet on the feature co-occurrence public dataset MAVD [23], we demonstrate its potential for extension to self-driving vehicle video-camera setups.

Figures (6)

• Figure 1: Experimental Setup. We positioned 6 microphones along the roadside and installed a webcam approximately 20 feet above the ground (3 of them are not shown in the picture). An ITS detecting idling vehicles displays a reminder message on the screen.
• Figure 2: Algorithm workflow. Our algorithm consists of an encoding module, a bidirectional attention module, and a region proposal network.
• Figure 3: Sample images from the AVIVD dataset, illustrating vehicles of various shapes, models, colors, and sizes. The dataset also features diverse lighting conditions.
• Figure 4: Dense Vehicle Trajectory Visualization. $X$ and $Y$ axes are aligned with image space. The rest axis is the time. Each 3D point represents the center of predicted bounding box. The color represents classes. Green is moving, red is idling, and blue is engine-off.
• Figure 5: IVD Visual Performance. The left image of each row shows the detected results (left) and the ground truth annotations (right). Green, red, and blue bounding boxes represent moving, idle, and non-idle vehicles respectively. The right image shows the corresponding spectrograms.