Correcting and Quantifying Systematic Errors in 3D Box Annotations for Autonomous Driving

TL;DR

This work addresses systematic errors in 3D box annotations caused by object motion across sensor timestamps in autonomous driving data. It introduces a motion-model-based offline correction framework that enforces physically plausible trajectories in the 2D BEV while remaining anchored to sensor detections, and it defines new metrics to quantify annotation quality. The method, evaluated on Argoverse 2, MAN TruckScenes, and a proprietary dataset, achieves over 17% improvement in annotation quality and reveals misplacements up to 2.5 m, predominantly along the travel direction, with speed amplifying the error. The findings show that annotation errors can significantly affect benchmark results, underscoring the importance of accurate, dynamic-aware annotations for reliable evaluation and downstream perception performance.

Abstract

Accurate ground truth annotations are critical to supervised learning and evaluating the performance of autonomous vehicle systems. These vehicles are typically equipped with active sensors, such as LiDAR, which scan the environment in predefined patterns. 3D box annotation based on data from such sensors is challenging in dynamic scenarios, where objects are observed at different timestamps, hence different positions. Without proper handling of this phenomenon, systematic errors are prone to being introduced in the box annotations. Our work is the first to discover such annotation errors in widely used, publicly available datasets. Through our novel offline estimation method, we correct the annotations so that they follow physically feasible trajectories and achieve spatial and temporal consistency with the sensor data. For the first time, we define metrics for this problem; and we evaluate our method on the Argoverse 2, MAN TruckScenes, and our proprietary datasets. Our approach increases the quality of box annotations by more than 17% in these datasets. Furthermore, we quantify the annotation errors in them and find that the original annotations are misplaced by up to 2.5 m, with highly dynamic objects being the most affected. Finally, we test the impact of the errors in benchmarking and find that the impact is larger than the improvements that state-of-the-art methods typically achieve with respect to the previous state-of-the-art methods; showing that accurate annotations are essential for correct interpretation of performance. Our code is available at https://github.com/alexandre-justo-miro/annotation-correction-3D-boxes.

Figures (4)

• Figure 1: Examples of 3D annotation errors in various datasets. The original points are colored by capture time offset relative to the reference time at which the boxes were annotated (see color bar at the right of the figure). The translucent, blue points show where these detections would have been at the annotation reference time. Our proposed annotations (dashed-line boxes) are aligned with points with zero time offset, while the original annotations (solid-line boxes) are aligned with point clouds with large time offsets. The position offset between them is an indication of the annotation error that we aim to correct.
• Figure 2: A diagram that illustrates the full optimization problem. The states to estimate, $X_i$, are within circles, whereas the variables that are used to compute the error are within squares. The motion model, $Y_i^{i+1}$, aims to enforce a physically plausible trajectory by minimizing the difference between the predicted states from the $i$th to the $(i+1)$th time step and the optimization variable at the $(i+1)$th time step. In turn, the point cloud, $P_i$, and distance to ego, $E_i$, error terms are applied at every time step and aim to place the corrected boxes coherently with the sensor data.
• Figure 3: Distributions of error of the original box annotations w.r.t. the corrected boxes in the selected sequences of the 3 examined datasets. Only boxes whose speed is at least [per-mode = symbol]3 are included in the plots.
• Figure 4: Box annotation error grouped by intervals. Within each interval, from left to right, the boxes represent the Argoverse 2 (blue), MAN TruckScenes (orange), and our proprietary (green) datasets, respectively. Within each box, from bottom to top, are represented: 5% percentile, 25% percentile, median, 75% percentile, and 95% percentile. Only objects whose speed is at least [per-mode = symbol]3 are included.