Mapping Semantic Segmentation to Point Clouds Using Structure from Motion for Forest Analysis

TL;DR

This work tackles the scarcity of publicly available, semantically labeled forest point clouds generated via Structure from Motion (SfM). It introduces a pipeline that combines a Unity-based forest simulator to produce labeled RGB imagery with ground-truth semantic masks and a modified OpenDroneMap workflow to propagate semantic labels into the resulting dense 3D point cloud. The semantic SfM enhancements include outlier rejection based on label consistency and per-point label and confidence estimation across image views, with results exported in the PLY format as label and confidence fields. On synthetic forest scenes, the approach yields coherent semantic segmentation in the reconstructed 3D data and establishes a foundation for training and evaluating models on semantically enriched forest SfM data, with planned extensions to real data and more advanced optimization methods. The method enables cost-effective, scalable semantic forest analysis beyond LiDAR, facilitating richer ecosystem understanding from image-derived reconstructions.

Abstract

Although the use of remote sensing technologies for monitoring forested environments has gained increasing attention, publicly available point cloud datasets remain scarce due to the high costs, sensor requirements, and time-intensive nature of their acquisition. Moreover, as far as we are aware, there are no public annotated datasets generated through Structure From Motion (SfM) algorithms applied to imagery, which may be due to the lack of SfM algorithms that can map semantic segmentation information into an accurate point cloud, especially in a challenging environment like forests. In this work, we present a novel pipeline for generating semantically segmented point clouds of forest environments. Using a custom-built forest simulator, we generate realistic RGB images of diverse forest scenes along with their corresponding semantic segmentation masks. These labeled images are then processed using modified open-source SfM software capable of preserving semantic information during 3D reconstruction. The resulting point clouds provide both geometric and semantic detail, offering a valuable resource for training and evaluating deep learning models aimed at segmenting real forest point clouds obtained via SfM.

Figures (7)

• Figure 1: Above: Frontal view of a generated forest sample scene. Below: Close up view.
• Figure 2: Samples from the imagery taken from the forest simulator. Left: RGB images of the generated forest scene; the red dots corresponds to the ground control points. Middle: Corresponding semantic segmentation image; each color is related to a different category: tree canopies (green), tree trunks (cyan), bushes, grass and understorey vegetation (yellow), ground (blue) and ground control points (red). Right: Greyscale version of the semantic segmentation image, for its use in the SfM algorithm.
• Figure 3: Point cloud obtained using the modified OpenDroneMaps, with the respective RGB color of each point.
• Figure 4: Semantic segmentation obtained for the point cloud of the synthetic forest scene. Each color represent a different category: ground (blue), tree trunks (yellow), tree canopies (green) and bushes, grass and understorey (orange).
• Figure 5: Close up view from two different perspectives of the semantic segmentation obtained. Each color represent a different category: ground (blue), tree trunks (yellow), tree canopies (green) and bushes, grass and understorey (orange).