Localization Is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix It

TL;DR

The paper reveals that popular online mapping datasets (nuScenes and Argoverse 2) suffer geographic data leakage between train, validation, and test splits, inflating reported performance. It introduces geographically disjoint Near and Far Extrapolation splits to measure true generalization to unseen areas and demonstrates substantial decreases in performance under these splits (e.g., up to $46.5$ mAP on nuScenes with MapTRv2). The authors re-evaluate state-of-the-art online-mapping methods, finding that conclusions about lifting strategies and auxiliary tasks shift when evaluated on fair geo-splits, with lifting and depth supervision often yielding smaller or different benefits. They provide publicly available splits and advocate for Far Extrapolation as a more realistic benchmark to drive genuine generalization in online mapping, highlighting the need for robust geo-aware evaluation in this domain.

Abstract

The task of online mapping is to predict a local map using current sensor observations, e.g. from lidar and camera, without relying on a pre-built map. State-of-the-art methods are based on supervised learning and are trained predominantly using two datasets: nuScenes and Argoverse 2. However, these datasets revisit the same geographic locations across training, validation, and test sets. Specifically, over $80$% of nuScenes and $40$% of Argoverse 2 validation and test samples are less than $5$ m from a training sample. At test time, the methods are thus evaluated more on how well they localize within a memorized implicit map built from the training data than on extrapolating to unseen locations. Naturally, this data leakage causes inflated performance numbers and we propose geographically disjoint data splits to reveal the true performance in unseen environments. Experimental results show that methods perform considerably worse, some dropping more than $45$ mAP, when trained and evaluated on proper data splits. Additionally, a reassessment of prior design choices reveals diverging conclusions from those based on the original split. Notably, the impact of lifting methods and the support from auxiliary tasks (e.g., depth supervision) on performance appears less substantial or follows a different trajectory than previously perceived. Splits can be found at https://github.com/LiljaAdam/geographical-splits

Figures (19)

• Figure 1: Example of substantial geographical overlap between train, val, and test sets for in nuScenes'. Green circle, Orange cross, and Red plus are training, validation, and test samples.
• Figure 2: Example from nuScenes where input images, predictions, and ground truth for a test sample are displayed together with the ground truth of the closest train sample. Lane dividers, road boundaries, and pedestrian crossings are visualized in orange, green, and blue.
• Figure 3: Number of samples within a cell with length $60$ m. Argoverse 2 has higher intra-set sample density.
• Figure 4: Predictions from MapTR liao2023maptr trained on Original and Geographical splits along with the ground truth. Yellow lines denote (lane) dividers, green (road) boundaries, and blue pedestrian crossings. Note that, when trained on the Original split, the branch to the parallel road on the left (teal box) is not visible in any image, yet appears in the predicted map. Also, the divider on the opposing road to the right (pink box) is predicted very well. When training on geographically split data (here Near Extrapolation), this method fails to predict these.
• Figure 5: In this example from Singapore Queenstown, the individual samples from a few sequences are divided into training (green) and validation (orange) according to the cut-off border. Some samples from a sequence are put in the training set, whereas the remaining are put in the validation set. The samples close to the dotted black cut-off line are the remaining possible data-leakage samples when using our proposed Near Extrapolation split.