WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments

TL;DR

This work introduces WildScenes, a bi-modal benchmark dataset consisting of multiple large-scale, sequential traversals in natural environments, including semantic annotations in high-resolution 2D images and dense 3D LiDAR point clouds, and accurate 6-DoF pose information.

Abstract

Recent progress in semantic scene understanding has primarily been enabled by the availability of semantically annotated bi-modal (camera and LiDAR) datasets in urban environments. However, such annotated datasets are also needed for natural, unstructured environments to enable semantic perception for applications, including conservation, search and rescue, environment monitoring, and agricultural automation. Therefore, we introduce $WildScenes$, a bi-modal benchmark dataset consisting of multiple large-scale, sequential traversals in natural environments, including semantic annotations in high-resolution 2D images and dense 3D LiDAR point clouds, and accurate 6-DoF pose information. The data is (1) trajectory-centric with accurate localization and globally aligned point clouds, (2) calibrated and synchronized to support bi-modal training and inference, and (3) containing different natural environments over 6 months to support research on domain adaptation. Our 3D semantic labels are obtained via an efficient, automated process that transfers the human-annotated 2D labels from multiple views into 3D point cloud sequences, thus circumventing the need for expensive and time-consuming human annotation in 3D. We introduce benchmarks on 2D and 3D semantic segmentation and evaluate a variety of recent deep-learning techniques to demonstrate the challenges in semantic segmentation in natural environments. We propose train-val-test splits for standard benchmarks as well as domain adaptation benchmarks and utilize an automated split generation technique to ensure the balance of class label distributions. The $WildScenes$ benchmark webpage is https://csiro-robotics.github.io/WildScenes, and the data is publicly available at https://data.csiro.au/collection/csiro:61541 .

Figures (13)

• Figure 1: The WildScenes benchmark consists of five large-scale traversals in two natural forest environments - Venman (V-01, V-02, V-03) and Karawatha (K-01, K-03). In the center of the figure, the traversals from each environment are depicted using the corresponding semantically annotated 3D global map of that traversal. The zoom-in views of example locations with prominent semantic classes are depicted. For each class example, three images depicting the 2D image, 2D semantic annotation, and 3D semantic point cloud of corresponding location and viewpoint are provided.
• Figure 2: The 3D semantic maps of the five traversals. The WildScenes contains repeat traversals of two natural environments, Venman (V-01, V-02, V-03) (left) and Karawatha (K-01, K-03) (right).
• Figure 3: Data collection campaign depicting the dense forest trails of Karawatha and Venman, respectively (left). The sensor payload comprises a spinning LiDAR sensor, encoder, IMU, GPS, and camera (right).
• Figure 4: Overview of the LabelCloud pipeline for generaing a 3D semantic map. We use Wildcat SLAM to calculate the trajectory and global map. Then, after annotating the 2D images, we perform label transfer from 2D images across multiple frames into 3D, utilizing the 6-DOF trajectory, to produce our 3D semantic point cloud.
• Figure 5: 2D (top) and 3D (bottom) label counts of WildScenes. The dashed line in the 2D counts represents the threshold for exclusion of a class for evaluation.