BEVal: A Cross-dataset Evaluation Study of BEV Segmentation Models for Autonomous Driving

TL;DR

A comprehensive cross-dataset evaluation of state-of-the-art BEV segmentation models to assess their performance across different training and testing datasets and setups, as well as different semantic categories, and investigates the influence of different sensors on the models' ability to generalize to diverse conditions and scenarios.

Abstract

Current research in semantic bird's-eye view segmentation for autonomous driving focuses solely on optimizing neural network models using a single dataset, typically nuScenes. This practice leads to the development of highly specialized models that may fail when faced with different environments or sensor setups, a problem known as domain shift. In this paper, we conduct a comprehensive cross-dataset evaluation of state-of-the-art BEV segmentation models to assess their performance across different training and testing datasets and setups, as well as different semantic categories. We investigate the influence of different sensors, such as cameras and LiDAR, on the models' ability to generalize to diverse conditions and scenarios. Additionally, we conduct multi-dataset training experiments that improve models' BEV segmentation performance compared to single-dataset training. Our work addresses the gap in evaluating BEV segmentation models under cross-dataset validation. And our findings underscore the importance of enhancing model generalizability and adaptability to ensure more robust and reliable BEV segmentation approaches for autonomous driving applications. The code for this paper available at https://github.com/manueldiaz96/beval .

Figures (6)

• Figure 1: Cross-dataset validation using the BEV semantic segmentation model LAPT-PP diaz2023laptnet. The left three columns show the Intersection Over Union (IoU) scores for three semantic categories when the model was trained on the Woven Planet dataset and tested on the nuScenes validation set (pink) and the Woven Planet validation set (gray). The right three columns show similar results when the model was trained on the nuScenes dataset initially. A significant performance drop is observed when the model is trained and tested on different datasets, highlighting its inherent limitations in generalization ability.
• Figure 2: Point cloud sample illustration (top) and histogram of the number of points per sample (bottom) for (a) nuScenes, (b) Woven Planet and (c) the subsampled Woven Planet point clouds. Best viewed with digital zoom.
• Figure 3: Example of map annotations provided by (a)nuScenes dataset and (b) Woven Planet dataset.
• Figure 4: Drivable Area ground truth generation for the Woven Planet Dataset. (a) Region of interest cropping. (b) Color filtering. (c) Gap filling and image resizing. Best viewed with digital zoom.
• Figure 5: Qualitative BEV semantic segmentation results for LAPT-PP on nuScenes dataset.