Ground Awareness in Deep Learning for Large Outdoor Point Cloud Segmentation

TL;DR

This work tackles semantic segmentation of large outdoor point clouds by introducing a non-local ground-awareness feature: the relative elevation $h_r$ to the terrain, derived from Digital Terrain Models. Implemented by augmenting RandLA-Net with $h_r$ (and other local features) and evaluated on three diverse datasets, the approach yields consistent improvements, most notably in Hessigheim where the average F1 rose from $72.35\%$ to $76.01\%$. The results demonstrate that non-local elevation context can compensate for limited receptive fields in dense point clouds, while the benefit of additional local features is dataset-dependent and can even hurt performance in highly variable density regimes. The authors advocate incorporating relative elevation in outdoor 3D segmentation tasks and suggest exploring its integration with self-supervised and Transformer-based models in future work.

Abstract

This paper presents an analysis of utilizing elevation data to aid outdoor point cloud semantic segmentation through existing machine-learning networks in remote sensing, specifically in urban, built-up areas. In dense outdoor point clouds, the receptive field of a machine learning model may be too small to accurately determine the surroundings and context of a point. By computing Digital Terrain Models (DTMs) from the point clouds, we extract the relative elevation feature, which is the vertical distance from the terrain to a point. RandLA-Net is employed for efficient semantic segmentation of large-scale point clouds. We assess its performance across three diverse outdoor datasets captured with varying sensor technologies and sensor locations. Integration of relative elevation data leads to consistent performance improvements across all three datasets, most notably in the Hessigheim dataset, with an increase of 3.7 percentage points in average F1 score from 72.35% to 76.01%, by establishing long-range dependencies between ground and objects. We also explore additional local features such as planarity, normal vectors, and 2D features, but their efficacy varied based on the characteristics of the point cloud. Ultimately, this study underscores the important role of the non-local relative elevation feature for semantic segmentation of point clouds in remote sensing applications.

Figures (4)

• Figure 1: View of the orthoprojection, DSM, DTM, NDSM, and 2D features (number of points $\nu$ and variance of the points $\sigma(z)$ in a “pixel” of an 2D grid) of the fifth Davos tile of the Swiss2DCities dataset. The Normalized DSM, or NDSM, is two-dimensional and calculated by subtracting the DTM from the DSM.
• Figure 2: The results on the fifth Davos validation tile of the Swiss3DCities dataset, with predictions of two configurations and corresponding ground truth. Some errors in Fig. \ref{['fig:davos_color']} that are not present in Fig. \ref{['fig:davos_all']} are circled in black.
• Figure 3: The test set of the Hessigheim March 2018 capture and predictions with different configurations in its entirety, and a zoomed in area. Errors in Fig. \ref{['fig:hessigheim_color']} are circled in red. Ground truth is not publicly available.
• Figure 4: A zoomed in area of the Toronto3D L002 validation tile with the predictions of two configurations and ground truth. The tile has been downsampled using grid sampling. An error in Fig. \ref{['fig:toronto_color']} is circled in red.