Fine-grained Metrics for Point Cloud Semantic Segmentation
Zhuheng Lu, Ting Wu, Yuewei Dai, Weiqing Li, Zhiyong Su
TL;DR
The paper tackles a core issue in point cloud semantic segmentation: evaluation metrics biased toward large or frequent categories. It introduces a suite of fine-grained metrics across four granularity levels—dataset-level, point-cloud-level, category-level, and instance-level—for both mean IoU and mean accuracy, incorporating NULL handling and an FP allocation scheme to address imbalances. The authors provide explicit definitions and formulas for $mIoU^{D}$, $mIoU^{P}$, $mIoU^{C}$, $mIoU^{I}$, $mAcc^{D}$, $mAcc^{P}$, $mAcc^{C}$, and $mAcc^{I}$, and validate them on ScanNet, S3DIS, and Semantic3D, showing that these metrics yield richer statistics and lessen bias toward large objects. The results demonstrate that relying on a single metric can mask weaknesses, and that reporting multiple fine-grained metrics enables more robust, nuanced comparisons, even in the absence of instance labels.
Abstract
Two forms of imbalances are commonly observed in point cloud semantic segmentation datasets: (1) category imbalances, where certain objects are more prevalent than others; and (2) size imbalances, where certain objects occupy more points than others. Because of this, the majority of categories and large objects are favored in the existing evaluation metrics. This paper suggests fine-grained mIoU and mAcc for a more thorough assessment of point cloud segmentation algorithms in order to address these issues. Richer statistical information is provided for models and datasets by these fine-grained metrics, which also lessen the bias of current semantic segmentation metrics towards large objects. The proposed metrics are used to train and assess various semantic segmentation algorithms on three distinct indoor and outdoor semantic segmentation datasets.