DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction

TL;DR

DreamCar addresses the challenge of reconstructing complete, high-quality 3D car models from in-the-wild imagery with limited viewpoints. It combines a car-specific generative prior learned from the Car360 dataset with symmetry-based data augmentation and a pose-optimization module to mitigate texture misalignment, integrating these with a multi-stage geometry pipeline guided by Score Distillation Sampling. The method demonstrates superior reconstruction quality and realism on Car360 and Nuscenes, outperforming NeRF-based and diffusion-based baselines and enabling scalable car asset generation for simulation and self-driving data augmentation. By curating Car360 and fusing 2D and 3D diffusion priors, DreamCar improves generalization to real-world cars and reduces reliance on manual asset creation, offering practical impact for large-scale vehicle asset libraries and realistic sensor simulation.

Abstract

Self-driving industries usually employ professional artists to build exquisite 3D cars. However, it is expensive to craft large-scale digital assets. Since there are already numerous datasets available that contain a vast number of images of cars, we focus on reconstructing high-quality 3D car models from these datasets. However, these datasets only contain one side of cars in the forward-moving scene. We try to use the existing generative models to provide more supervision information, but they struggle to generalize well in cars since they are trained on synthetic datasets not car-specific. In addition, The reconstructed 3D car texture misaligns due to a large error in camera pose estimation when dealing with in-the-wild images. These restrictions make it challenging for previous methods to reconstruct complete 3D cars. To address these problems, we propose a novel method, named DreamCar, which can reconstruct high-quality 3D cars given a few images even a single image. To generalize the generative model, we collect a car dataset, named Car360, with over 5,600 vehicles. With this dataset, we make the generative model more robust to cars. We use this generative prior specific to the car to guide its reconstruction via Score Distillation Sampling. To further complement the supervision information, we utilize the geometric and appearance symmetry of cars. Finally, we propose a pose optimization method that rectifies poses to tackle texture misalignment. Extensive experiments demonstrate that our method significantly outperforms existing methods in reconstructing high-quality 3D cars. \href{https://xiaobiaodu.github.io/dreamcar-project/}{Our code is available.}

Figures (7)

• Figure 1: The illustration of the application scenario of our work and comparison with other methods.a. Utilizing merely 4 reference images even less in the moving forward scene, we achieve the reconstruction of a complete 3D object, which is then simulated within a realistic scene. b. The novel view synthesis comparison provided by Zero-123-XL gradually trained on our Car360 dataset. c. The visual comparison of different 3D reconstruction methods.
• Figure 2: The illustration of different car datasets.a. The comparison of the total number of vehicles across different datasets, highlights that our Car360 collection has the highest count, with 5,600 vehicles. b. The visual comparison of different car datasets. c. The illustration of the distribution of various vehicle categories, lighting conditions, and the number of captured views in our collected Car360 dataset.
• Figure 3: The illustration of our DreamCar. Our method can be divided into geometry reconstruction and texture refinement stages. We input reference views (original reference views and their mirror counterparts) with generative prior guiding our 3D model to reconstruct 3D cars in all stages. In the geometry reconstruction stage, our method progressively sculpts fine geometry with coarse texture. In the texture refinement stage, we focus on the refinement of the appearance of the car with the DreamBooth technique.
• Figure 4: Qualitative evaluation of 3D reconstruction on the Nuscenes Dataset nuscenes. The renderings are provided from various viewpoints distinct from the reference image, illustrating the completeness of the reconstructed 3D models.
• Figure 5: The ablation study of our proposed method. Given the leftmost references, we ablate the mirror and pose optimization techniques to demonstrate our method.