OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective

TL;DR

OccuFly addresses the lack of real-world aerial SSC benchmarks by introducing a camera-based data-generation pipeline that converts geo-referenced imagery into dense 3D semantic grids via SfM/MVS and 2D-to-3D label transfer. The dataset comprises nine scenes, over $20{,}000$ samples, $22$ semantic classes, and per-frame metric depth maps, all organized in SSCBench format, enabling scalable evaluation of aerial SSC methods. It demonstrates a baseline evaluation with CGFormer and a monocular depth estimator adapted to OccuFly (DAv2-OccuFly), revealing altitude-dependent performance and domain-specific challenges for aerial SSC. The work provides a substantial resource for holistic aerial 3D scene understanding using camera data, highlighting opportunities for dynamic scene handling, automated labeling, and improved depth priors to advance practical UAV perception.

Abstract

Semantic Scene Completion (SSC) is crucial for 3D perception in mobile robotics, as it enables holistic scene understanding by jointly estimating dense volumetric occupancy and per-voxel semantics. Although SSC has been widely studied in terrestrial domains such as autonomous driving, aerial scenarios like autonomous flying remain largely unexplored, thereby limiting progress on downstream applications. Furthermore, LiDAR sensors represent the primary modality for SSC data generation, which poses challenges for most uncrewed aerial vehicles (UAVs) due to flight regulations, mass and energy constraints, and the sparsity of LiDAR-based point clouds from elevated viewpoints. To address these limitations, we introduce OccuFly, the first real-world, camera-based aerial SSC benchmark, captured at altitudes of 50m, 40m, and 30m during spring, summer, fall, and winter. OccuFly covers urban, industrial, and rural scenarios, provides 22 semantic classes, and the data format adheres to established conventions to facilitate seamless integration with existing research. Crucially, we propose a LiDAR-free data generation framework based on camera modality, which is ubiquitous on modern UAVs. By utilizing traditional 3D reconstruction, our framework automates label transfer by lifting a subset of annotated 2D masks into the reconstructed point cloud, thereby substantially minimizing manual 3D annotation effort. Finally, we benchmark the state-of-the-art on OccuFly and highlight challenges specific to elevated viewpoints, yielding a comprehensive vision benchmark for holistic aerial 3D scene understanding.

Figures (10)

• Figure 1: OccuFly introduces the first real-world, aerial 3D SSC benchmark dataset, consisting of $9$ scenes that provide over 20000 samples of RGB images, semantic occupancy grids, and metric depth maps, including $22$ semantic classes. OccuFly covers almost 200000m2 at 50m, 40m, and 30m altitude in urban, industrial, and rural scenarios during spring, summer, fall, and winter. Zoom in for best view.
• Figure 2: Proposed image-based data generation framework. An overview is provided in \ref{['subsec_overview']}. Zoom in for best view.
• Figure 3: Quantitative results on OccuFly test set. We evaluate CGFormer cgformer (i) class-wise, (ii) group-wise, and (iii) altitude-wise, effectively investigating implications on key characteristics that are natural to image-based aerial data generation and learning for SSC.
• Figure 4: Qualitative visualization results on the OccuFly test set (\ref{['subsec_dataset_statistics']}).
• Figure 5: Qualitative evaluation results of DAv2-OccuFly (\ref{['subsec_experimental_setup']}) on the OccuFly test set (see \ref{['subsec_dataset_statistics']}). Zoom in for best view.