Empir3D : A Framework for Multi-Dimensional Point Cloud Assessment

TL;DR

Empir3D tackles the limitation of uni-dimensional point-cloud quality metrics by introducing four independent, region-aware metrics: $Q_r$ (resolution), $Q_a$ (accuracy), $Q_c$ (coverage), and $Q_t$ (artifact score). It formalizes a region-based evaluation framework and demonstrates its usefulness through ablations and applications to dense SLAM and learning-based point-cloud completion, revealing richer diagnostics than Chamfer, Hausdorff, or Earth-Mover distances. The framework achieves practical impact by enabling more nuanced debugging, sensor characterization, and method comparison for large-scale point clouds, with scalable compute ($O(n \log n)$) and a path toward open-source release. Overall, Empir3D provides a disciplined, multi-dimensional lens for understanding and improving 3D perception pipelines in real-world settings.

Abstract

Advancements in sensors, algorithms, and compute hardware have made 3D perception feasible in real time. Current methods to compare and evaluate the quality of a 3D model, such as Chamfer, Hausdorff, and Earth-Mover's distance, are uni-dimensional and have limitations, including an inability to capture coverage, local variations in density and error, and sensitivity to outliers. In this paper, we propose an evaluation framework for point clouds (Empir3D) that consists of four metrics: resolution to quantify the ability to distinguish between individual parts in the point cloud, accuracy to measure registration error, coverage to evaluate the portion of missing data, and artifact score to characterize the presence of artifacts. Through detailed analysis, we demonstrate the complementary nature of each of these dimensions and the improvements they provide compared to the aforementioned uni-dimensional measures. Furthermore, we illustrate the utility of Empir3D by comparing our metrics with uni-dimensional metrics for two 3D perception applications (SLAM and point cloud completion). We believe that Empir3D advances our ability to reason about point clouds and helps better debug 3D perception applications by providing a richer evaluation of their performance. Our implementation of Empir3D, custom real-world datasets, evaluations on learning methods, and detailed documentation on how to integrate the pipeline will be made available upon publication.

Figures (8)

• Figure 1: Figure demonstrates cells of size $\epsilon$, green cells contain both ground truth (green) and candidate (blue) points making them covered, red cells only contain candidate points making them artifacts and grey with only ground truth points showing missing coverage (or un-covered). Top row 3rd cell shows $d<\epsilon$ as the distance considered to compute accuracy based on \ref{['accr']}
• Figure 2: Ablation Study on street block dataset; Left to right: Down-sampled, Noise Added, Cropped Simulated Artifacts
• Figure 3: Simulation dataset; Point cloud built using FAST-LIO2 (Top) and LeGO-LOAM (Bottom). Zoomed in view for qualitative assessment
• Figure 4: Real-world evaluation of Dense SLAM - Point clouds map generated using FAST-LIO2 (Spot robot + Ouster OS-1 128 LiDAR) on the left, and ground truth on the right ( robotic total-station).
• Figure 5: Evaluation on Davis dataset, zoomed-in view shows variations in detail for different SLAM methods. Top to Bottom: FAST-LIO2, Ground Truth, LeGO-LOAM. Zoomed in view of staircase on the right for qualitative assessment