What's Wrong with the Absolute Trajectory Error?

TL;DR

This paper tackles the fragility of Absolute Trajectory Error (ATE) in the presence of outliers by introducing Discernible Trajectory Error (DTE) and Discernible Rotation Error (DRE), which use median-based alignment, geodesic medians on $SO(3)$, MAD-based scaling, and winsorization to robustly quantify trajectory and rotation accuracy. The final DTE and DRE combine robust mean and RMS components to reflect both inlier accuracy and outlier prevalence, enabling clearer discrimination as noise or outliers vary. A simple calibration procedure for the camera-to-marker rotation is proposed and analyzed, with extensive simulations and real-data validation showing that DTE/DRE offer more informative evaluation than ATE in challenging scenarios. The work also provides practical insights into parameter choices and discusses limitations and future directions for robust trajectory evaluation.

Abstract

One of the limitations of the commonly used Absolute Trajectory Error (ATE) is that it is highly sensitive to outliers. As a result, in the presence of just a few outliers, it often fails to reflect the varying accuracy as the inlier trajectory error or the number of outliers varies. In this work, we propose an alternative error metric for evaluating the accuracy of the reconstructed camera trajectory. Our metric, named Discernible Trajectory Error (DTE), is computed in five steps: (1) Shift the ground-truth and estimated trajectories such that both of their geometric medians are located at the origin. (2) Rotate the estimated trajectory such that it minimizes the sum of geodesic distances between the corresponding camera orientations. (3) Scale the estimated trajectory such that the median distance of the cameras to their geometric median is the same as that of the ground truth. (4) Compute, winsorize and normalize the distances between the corresponding cameras. (5) Obtain the DTE by taking the average of the mean and the root-mean-square (RMS) of the resulting distances. This metric is an attractive alternative to the ATE, in that it is capable of discerning the varying trajectory accuracy as the inlier trajectory error or the number of outliers varies. Using the similar idea, we also propose a novel rotation error metric, named Discernible Rotation Error (DRE), which has similar advantages to the DTE. Furthermore, we propose a simple yet effective method for calibrating the camera-to-marker rotation, which is needed for the computation of our metrics. Our methods are verified through extensive simulations.

Figures (7)

• Figure 1: Reference frames that are relevant for describing the camera pose.
• Figure 2: Each of the colored blocks represents the normalized ATE and DTE obtained with the given number of outliers and noise level in camera positions. In the presence of outliers, the DTE shows a more pronounced gradation than the ATE. This means that it is better at capturing the varying trajectory accuracy as the number of outliers and the noise level varies.
• Figure 3: [Left] The errors corresponding to each column of Fig. \ref{['fig:ATE_vs_DTE']}, showing the effect of a varying number of outliers. We shift each column values such that the error in the outlier-free case is 0. As the number of outliers increases, the ATE curves flatten, which signals a decreasing level of sensitivity. In contrast, the DTE maintains a relatively high level of sensitivity. [Right] The errors corresponding to each row of Fig. \ref{['fig:ATE_vs_DTE']}, showing the effect of a varying noise level in camera positions. We shift each row values such that the error in the zero-noise case is 0. As the number of outliers increases, the slope of the ATE curve drastically decreases, and with just three outliers, it becomes almost insensitive to the noise level. In contrast, the DTE can still maintain a moderate level of sensitivity, even with 10 outliers.
• Figure 4: Each of the colored blocks represents the normalized mean and RMS trajectory errors obtained with the given number of outliers and noise level in camera positions. The mean error shows a high level of sensitivity to both the number of outliers and the noise level, whereas the RMS error is sensitive to the number of outliers, but not so much to the noise level.
• Figure 5: Each of the colored blocks represents the rotation error obtained with the given number of outliers and noise level in camera orientations. Compared to the RMS errors, the median and mean errors have relatively low sensitivity to the number of outliers and relatively high sensitivity to the noise level. The DRE is sufficiently sensitive to the noise level, but more to the number of outliers, which is a desirable property.