3DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Views

TL;DR

The paper addresses the lack of large-scale real-world 3D car datasets by introducing 3DRealCar, a dataset with 2,500 real cars and an average of ~200 dense RGB-D 360-degree views per car, captured under three lighting conditions and accompanied by 13-class car parsing maps. It details data collection via ARKit, background removal, orientation normalization, and size rescaling to enable high-fidelity 3D reconstruction. The authors benchmark a range of 2D and 3D tasks, including neural-field and diffusion-based 3D generation, and demonstrate both the usefulness of the dataset for improving real-car priors and the challenges posed by reflective/dark lighting. The work provides extensive baselines and highlights directions for future work, such as interior-view data and more robust reconstructions under challenging lighting.

Abstract

3D cars are commonly used in self-driving systems, virtual/augmented reality, and games. However, existing 3D car datasets are either synthetic or low-quality, limiting their applications in practical scenarios and presenting a significant gap toward high-quality real-world 3D car datasets. In this paper, we propose the first large-scale 3D real car dataset, termed 3DRealCar, offering three distinctive features. (1) \textbf{High-Volume}: 2,500 cars are meticulously scanned by smartphones, obtaining car images and point clouds with real-world dimensions; (2) \textbf{High-Quality}: Each car is captured in an average of 200 dense, high-resolution 360-degree RGB-D views, enabling high-fidelity 3D reconstruction; (3) \textbf{High-Diversity}: The dataset contains various cars from over 100 brands, collected under three distinct lighting conditions, including reflective, standard, and dark. Additionally, we offer detailed car parsing maps for each instance to promote research in car parsing tasks. Moreover, we remove background point clouds and standardize the car orientation to a unified axis for the reconstruction only on cars and controllable rendering without background. We benchmark 3D reconstruction results with state-of-the-art methods across different lighting conditions in 3DRealCar. Extensive experiments demonstrate that the standard lighting condition part of 3DRealCar can be used to produce a large number of high-quality 3D cars, improving various 2D and 3D tasks related to cars. Notably, our dataset brings insight into the fact that recent 3D reconstruction methods face challenges in reconstructing high-quality 3D cars under reflective and dark lighting conditions. \textcolor{red}{\href{https://xiaobiaodu.github.io/3drealcar/}{Our dataset is here.}}

Figures (9)

• Figure 1: Characteristics of our curated high-quality 3DRealCar dataset. 3DRealCar contains detailed annotations for various colors, car types, brands, and car parsing maps. 3DRealCar contains three lighting conditions on car surfaces, bringing challenges to existing methods.
• Figure 2: Visual comparisons of 3D car datasets and the results of a 3D generative method. Our 3DRealCar is captured in real-world scenes and contains more densely captured views. In addition, our dataset has annotations for three different lighting conditions on the car surface. We also compare a recent state-of-the-art text-to-3D model, MVDream shi2023mvdream with a prompt "a modern sedan", demonstrating its failure to generate high-quality 3D car models.
• Figure 3: The applicable tasks of our dataset. Our proposed 3DRealCar dataset containing RGB-D images, point clouds, and rich annotations, can be applied to various popular 2D and 3D tasks to support the construction of safe and reliable self-driving systems.
• Figure 4: Illustration of our data collection and preprocessing. We first circle a car three times while scanning the car with a smartphone for the attainment of RGB-D images and its point clouds. Then we use Colmap sfm2016 and SAM sam to obtain poses and remove the background point clouds. Finally, we use the 3DGS kerbl2023gaussiansplatting trained on the processed data to obtain 3D car model.
• Figure 5: The distributions of our 3DRealCar dataset. We show distributions of car types, lighting conditions, captured views, car colors, and car size. We try our best to capture cars with various colors and types for the diversity of our dataset.