Collision Avoidance Metric for 3D Camera Evaluation

TL;DR

This work tackles the gap between traditional point-cloud metrics and downstream collision-avoidance performance by introducing the Collision Avoidance metric, a simulation-based evaluation that uses a rectangular gripper to compare predicted versus ground-truth collisions across multiple directions. It defines actionable quantities—false positive collision rate $R_{FPC}$, false negative collision rate $R_{FNC}$, and Collision F-score $FC$—and supports tunable tolerances and directionality to reflect real-world safety margins. Through test scenes combining active/passive stereo and structured-light sensing, the method demonstrates how the metric yields sensor rankings that align with downstream collision avoidance better than conventional metrics, while providing interpretable failure modes via visualized collision paths. The approach offers a practical framework for sensor selection and parameter tuning in robotics and autonomous driving applications where safe navigation is paramount.

Abstract

3D cameras have emerged as a critical source of information for applications in robotics and autonomous driving. These cameras provide robots with the ability to capture and utilize point clouds, enabling them to navigate their surroundings and avoid collisions with other objects. However, current standard camera evaluation metrics often fail to consider the specific application context. These metrics typically focus on measures like Chamfer distance (CD) or Earth Mover's Distance (EMD), which may not directly translate to performance in real-world scenarios. To address this limitation, we propose a novel metric for point cloud evaluation, specifically designed to assess the suitability of 3D cameras for the critical task of collision avoidance. This metric incorporates application-specific considerations and provides a more accurate measure of a camera's effectiveness in ensuring safe robot navigation. The source code is available at https://github.com/intrinsic-ai/collision-avoidance-metric.

Figures (8)

• Figure 1: Visualization of the Collision Avoidance Metric evaluation process. A set of grippers moving towards the point cloud is simulated with the same initial direction vector sampled on the XY plane with step S until collision is detected.
• Figure 2: The process of detecting a False Positive Collision on a gripper's path. A collision is initially detected if the number of points inside the gripper $N_c$ is more than a threshold number of points, denoted by $N_{GT}$ for the Ground Truth point cloud and $N_Q$ for the Query point cloud. A collision is considered false positive if: 1. The Query point cloud collides with the gripper before the GT point cloud. 2. The distance between the two collision points is greater than the Z tolerance.
• Figure 3: A sample scene from the evaluation dataset. Collision maps are aligned with each point cloud. Red dots indicate FNC paths: collisions missed by the captured point cloud, while blue dots represent FPC paths: "ghost" collisions detected by the captured point cloud. Note that the CREStereo Active has lower Chamfer and Hausdorff distances than CREStereo Passive, but it also has significantly higher $R_{FNC}$ and $FC$. These metrics alone would not reveal the increased risk of collision for the robot gripper in this scene. Similarly, Photoneo achieves a better F-score and lower Chamfer and Hausdorff distances compared to CREStereo Passive, but performs slightly worse in $FC$ again, primarily due to its higher $R_{FNC}$. For optimal viewing, zoom in on a digital copy of the paper.
• Figure 4: A crop of a point cloud with simulated paths. Black dots are the paths with Aligned label from \ref{['subsec:algorithm']}, blue dots - the FPC paths, red dots - the FNC paths.
• Figure 5: The red dots represent FNC points, indicating missed collisions during a specific gripper movement direction. Capturing metallic surfaces, like this pipe, can be challenging for structured light 3D scanners.