WHU-Synthetic: A Synthetic Perception Dataset for 3-D Multitask Model Research

TL;DR

WHU-Synthetic introduces a large-scale synthetic 3D dataset designed for multitask learning, aligning depth completion, upsampling, segmentation, place recognition, and 3D reconstruction within the same environmental domain. It incorporates novel sensor configurations (multi-density LiDAR), city-scale surface sampling, and temporal changes to enable robust cross-task exploration and domain adaptation studies. Baseline experiments validate dataset utility and reveal task-specific gains and limitations, while multi-task explorations demonstrate mutual benefits among sub-tasks when sharing a backbone and joint losses. The work highlights the potential of synthetic data to advance 3D multitask perception, with practical implications for unified models and robust scene understanding, and discusses domain gaps and future directions for expanding task coverage.

Abstract

End-to-end models capable of handling multiple sub-tasks in parallel have become a new trend, thereby presenting significant challenges and opportunities for the integration of multiple tasks within the domain of 3D vision. The limitations of 3D data acquisition conditions have not only restricted the exploration of many innovative research problems but have also caused existing 3D datasets to predominantly focus on single tasks. This has resulted in a lack of systematic approaches and theoretical frameworks for 3D multi-task learning, with most efforts merely serving as auxiliary support to the primary task. In this paper, we introduce WHU-Synthetic, a large-scale 3D synthetic perception dataset designed for multi-task learning, from the initial data augmentation (upsampling and depth completion), through scene understanding (segmentation), to macro-level tasks (place recognition and 3D reconstruction). Collected in the same environmental domain, we ensure inherent alignment across sub-tasks to construct multi-task models without separate training methods. Besides, we implement several novel settings, making it possible to realize certain ideas that are difficult to achieve in real-world scenarios. This supports more adaptive and robust multi-task perception tasks, such as sampling on city-level models, providing point clouds with different densities, and simulating temporal changes. Using our dataset, we conduct several experiments to investigate mutual benefits between sub-tasks, revealing new observations, challenges, and opportunities for future research. The dataset is accessible at https://github.com/WHU-USI3DV/WHU-Synthetic.

Figures (14)

• Figure 1: Comparison between dataset quantities in 2D and 3D vision, as well as the size of some classic datasets. The statistic data is from the Papers With Code paperswithcode website.
• Figure 2: The distinctive features of WHU-Synthetic dataset. We implemented many novel settings, such as the stacking of sensors and the generation of annotated city mesh models. These attributes enable us to realize multi-task functionalities and data that are challenging to achieve in the real world.
• Figure 3: The vehicle system and the sensor layout. We successfully stack LiDARs with different channel counts at the same location to simulate point clouds of different densities at the sensor level, and the data are automatically annotated. The point clouds generated by LiDAR sensors share the same coordinate system and do not produce mutual occlusions.
• Figure 4: Depth completion results of PENet from different input (32/64/128-channel LiDAR projected image).
• Figure 5: Domain adaptation visualization results of upsampling on ILN.