Reformulating AI-based Multi-Object Relative State Estimation for Aleatoric Uncertainty-based Outlier Rejection of Partial Measurements

TL;DR

This work tackles robust object-relative state estimation for mobile robots by reformulating the EKF update to use direct object-relative $6$-DoF measurements, which decouples translation and rotation and enables partial measurement rejection. It replaces fixed DL-pose covariances with dynamically predicted aleatoric uncertainty, guiding both the update and outlier rejection, and introduces partial measurement rejection (AORP) to maintain usable information when rotations are ambiguous. Empirical results on synthetic YCB-V data show that the direct measurement EKF with aleatoric-based covariance and AORP outperforms the inverse-measurement approach in accuracy and consistency, and reduces divergence under challenging viewpoints. The proposed approach reduces the need for manual covariance engineering and enhances robustness of AI-based perception in real-time, edge-deployed robotic systems.

Abstract

Precise localization with respect to a set of objects of interest enables mobile robots to perform various tasks. With the rise of edge devices capable of deploying deep neural networks (DNNs) for real-time inference, it stands to reason to use artificial intelligence (AI) for the extraction of object-specific, semantic information from raw image data, such as the object class and the relative six degrees of freedom (6-DoF) pose. However, fusing such AI-based measurements in an Extended Kalman Filter (EKF) requires quantifying the DNNs' uncertainty and outlier rejection capabilities. This paper presents the benefits of reformulating the measurement equation in AI-based, object-relative state estimation. By deriving an EKF using the direct object-relative pose measurement, we can decouple the position and rotation measurements, thus limiting the influence of erroneous rotation measurements and allowing partial measurement rejection. Furthermore, we investigate the performance and consistency improvements for state estimators provided by replacing the fixed measurement covariance matrix of the 6-DoF object-relative pose measurements with the predicted aleatoric uncertainty of the DNN.

Figures (6)

• Figure 1: Comparison of the 6-DoF object-relative pose measurements (blue) influence on the state estimation task (gray) for the proposed direct measurement and the inverse measurement jantos2023aiobjrelstate approaches. In both cases, the 6-DoF object pose is initially measured with respect to the mobile robot. By inverting the measurement, the relative pose measurement is expressed in the estimated object frame, thus rotation measurement errors lead to discrepancies between the measured and estimated mobile robot 6-DoF pose. In contrast, the proposed direct approach shifts the measurement error towards the object's estimate rather than the mobile robot's state. Moreover, our approach enables the decoupling of the translation and rotation measurement, allowing the rejection of either part.
• Figure 2: Visualization of the different reference frames. The goal is to estimate the position and orientation of a rigid body (red) consisting of an IMU ($\mathsf{I}$) and camera ($\mathsf{C}$) in a fixed but arbitrary navigation frame ($\mathsf{W}$) with respect to a set of objects of interest ($\mathsf{O_i}$, blue). A DL-based pose predictor provides the object-relative 6-DoF pose measurements to the state estimator (green). The extrinsic calibration between the IMU and the camera is an additional auxiliary state in our formulation.
• Figure 3: Left: Example of a synthetic training image containing the subset of the YCB-V objects and distractor objects. Right: Example image from an evaluation trajectory with a challenging object constellation and occlusion.
• Figure 4: Example trajectory used for evaluation with varying distances and viewing angles of the objects.
• Figure 5: Direct Measurement (ours): Comparison of the absolute translation error (left, red) and rotation error (right, red) to the estimated aleatoric uncertainty (black) across the whole trajectory for the mug. Note the different scales in the plots' axes and across the plots.