PLoc: A New Evaluation Criterion Based on Physical Location for Autonomous Driving Datasets

TL;DR

The paper identifies a key gap in autonomous driving evaluation: the neglect of objects' physical locations and their safety relevance. It introduces PLoc, a location-weighted evaluation criterion, and the ApolloScape-R dataset (ApolloScape-R) with pedestrians re-annotated by on-road vs sidewalk status to test this criterion using ten detectors. Experimental results show pedestrians in the travel lane are consistently harder to detect than those on sidewalks, underscoring the need for location-aware metrics and training adaptations. The approach has practical impact by improving safety-oriented evaluation and can be extended to other objects and signaling elements in autonomous driving contexts.

Abstract

Autonomous driving has garnered significant attention as a key research area within artificial intelligence. In the context of autonomous driving scenarios, the varying physical locations of objects correspond to different levels of danger. However, conventional evaluation criteria for automatic driving object detection often overlook the crucial aspect of an object's physical location, leading to evaluation results that may not accurately reflect the genuine threat posed by the object to the autonomous driving vehicle. To enhance the safety of autonomous driving, this paper introduces a novel evaluation criterion based on physical location information, termed PLoc. This criterion transcends the limitations of traditional criteria by acknowledging that the physical location of pedestrians in autonomous driving scenarios can provide valuable safety-related information. Furthermore, this paper presents a newly re-annotated dataset (ApolloScape-R) derived from ApolloScape. ApolloScape-R involves the relabeling of pedestrians based on the significance of their physical location. The dataset is utilized to assess the performance of various object detection models under the proposed PLoc criterion. Experimental results demonstrate that the average accuracy of all object detection models in identifying a person situated in the travel lane of an autonomous vehicle is lower than that for a person on a sidewalk. The dataset is publicly available at https://github.com/lnyrlyed/ApolloScape-R.git

Figures (3)

• Figure 1: Four images are from the ApolloScape-R dataset. The red bounding box marked people on the road, who are in the path of the driving car, a potential danger position and need to be given more weight. The green bounding box marked people on the sidewalk, who are in a safe position and pose almost no threat to the driving car.
• Figure 2: The ApolloScape huang2018apolloscape dataset has some images that contain only vehicles, the two images on the left are examples from the ApolloScape huang2018apolloscape dataset, whereas we focus more on pedestrians as objects. Therefore, the images containing people were selected and re-annotated to form the new ApolloScape-R dataset. People in different physical locations are separated into two categories. Two images in the middle is labeled with a green bounding box. Pedestrians are on the sidewalk and pose little threat to the autonomous driving cars. On the right images, people who labeled with a red bounding box are in the direction of autonomous driving cars and are more likely to be involved in an accident.
• Figure 3: The tree on the left is in the middle of the vehicle's driving path and can directly affect driving safety. Autonomous driving vehicles need to accurately recognize trees to avoid making mistakes that could lead to danger. The tree on the right appears only as a background and does not pose a threat to the autonomous driving vehicle's decisions.