Real-Time Idling Vehicles Detection using Combined Audio-Visual Deep Learning

TL;DR

A real-time, dynamic vehicle idling detection algorithm that can detect engine switching on or off instantly and achieves 71.02 average precision for idle detections and 91.06 for engine off detections is presented.

Abstract

Combustion vehicle emissions contribute to poor air quality and release greenhouse gases into the atmosphere, and vehicle pollution has been associated with numerous adverse health effects. Roadways with extensive waiting and/or passenger drop off, such as schools and hospital drop-off zones, can result in high incidence and density of idling vehicles. This can produce micro-climates of increased vehicle pollution. Thus, the detection of idling vehicles can be helpful in monitoring and responding to unnecessary idling and be integrated into real-time or off-line systems to address the resulting pollution. In this paper we present a real-time, dynamic vehicle idling detection algorithm. The proposed idle detection algorithm and notification rely on an algorithm to detect these idling vehicles. The proposed method relies on a multi-sensor, audio-visual, machine-learning workflow to detect idling vehicles visually under three conditions: moving, static with the engine on, and static with the engine off. The visual vehicle motion detector is built in the first stage, and then a contrastive-learning-based latent space is trained for classifying static vehicle engine sound. We test our system in real-time at a hospital drop-off point in Salt Lake City. This in-situ dataset was collected and annotated, and it includes vehicles of varying models and types. The experiments show that the method can detect engine switching on or off instantly and achieves 71.02 average precision (AP) for idle detections and 91.06 for engine off detections.

Figures (6)

• Figure 1: Proposed System Design. The yellow arrow collects vehicle motion, engine sound, and pollutant concentrations. The red arrow represents data transmission to the computer. The green arrow denotes sending the predicted idling status to the displays. The blue arrow represents the driver receiving the information from the display and potentially making behavior changes.
• Figure 2: (a) Our IVD Algorithm. (b) Class Definition Hierarchy.
• Figure 3: System Setup. Wireless microphones are stuck on the right wall. The camera is mounted on a tripod near rocks.
• Figure 4: MDS 2D Visualization on Audio Encoder Latent Space. Due to the huge amount of validation samples, we feed part of our validation data into trained SCL and SL's encoder. The left side shows projected latent space of SCL in 2D dimension. The right side shows SL's latent space. Red dots are background samples. Blue dots are foreground samples.
• Figure 5: IVD Visual Performance. Each row includes detected results and groundtruth annotations along with the corresponding spectrogram. Red, green, and blue bounding boxes are idle, non-idle, and moving labels respectively. Dotted and solid rectangles are prediction and ground truth.