ETHcavation: A Dataset and Pipeline for Panoptic Scene Understanding and Object Tracking in Dynamic Construction Environments

TL;DR

ETHcavation presents a unified panoptic scene understanding system tailored for construction sites by fusing 2D panoptic segmentation with 3D LiDAR mapping and dynamic object tracking. It introduces a two-layer semantic map, a Kalman-tracking pipeline for dynamic objects, and a fine-tuning strategy using a small, domain-specific dataset, including a public 502-image construction-site dataset and released code. The approach is validated through offline segmentation metrics and online navigation with an online RRT* planner, demonstrating robust, real-time reactive path planning in dynamic, unstructured environments. The work provides practical contributions for perception-driven autonomous construction-site robotics and offers resources to accelerate future research in this domain.

Abstract

Construction sites are challenging environments for autonomous systems due to their unstructured nature and the presence of dynamic actors, such as workers and machinery. This work presents a comprehensive panoptic scene understanding solution designed to handle the complexities of such environments by integrating 2D panoptic segmentation with 3D LiDAR mapping. Our system generates detailed environmental representations in real-time by combining semantic and geometric data, supported by Kalman Filter-based tracking for dynamic object detection. We introduce a fine-tuning method that adapts large pre-trained panoptic segmentation models for construction site applications using a limited number of domain-specific samples. For this use case, we release a first-of-its-kind dataset of 502 hand-labeled sample images with panoptic annotations from construction sites. In addition, we propose a dynamic panoptic mapping technique that enhances scene understanding in unstructured environments. As a case study, we demonstrate the system's application for autonomous navigation, utilizing real-time RRT* for reactive path planning in dynamic scenarios. The dataset (https://leggedrobotics.github.io/panoptic-scene-understanding.github.io/) and code (https://github.com/leggedrobotics/rsl_panoptic_mapping) for training and deployment are publicly available to support future research.

Figures (7)

• Figure 1: Autonomous navigation with the M545 excavator. The top image illustrates the model's 2D panoptic segmentation prediction, while the bottom image depicts the panoptic map and the planned path of the navigation planner in black.
• Figure 2: Overview of the integrated camera and LiDAR data processing and semantic mapping pipeline for autonomous navigation in unstructured environments.
• Figure 3: Exemplary samples from the generated and hand-labeled dataset. The 502 images published in this work are recorded in various environments, including real-world construction sites, road navigation, and natural environments.
• Figure 4: Segmentation of point cloud data into dynamic (red) and static (blue) elements, highlighting dynamic object tracking (e.g., people, unidentified objects) as bounding boxes.
• Figure 5: Two exemplary segmentation masks produced by Mask2Former.