RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image

TL;DR

This work addresses automatic generation of high-fidelity 3D car assets from a single image, enabling relighting under arbitrary illumination. It introduces Carverse, a large-scale synthetic dataset with geometry and material maps, and Relightable 3D-GS (RGM), a feed-forward model that outputs relightable Gaussian primitives and global illumination. Through a physically-based rendering layer and Disney BRDFs, RGM recovers geometry, albedo, roughness, and metallic properties while supporting realistic relighting. Experiments show state-of-the-art performance on Carverse and a practical autonomous driving simulation pipeline that inserts relighted cars into road scenes with photorealistic rendering.

Abstract

The generation of high-quality 3D car assets is essential for various applications, including video games, autonomous driving, and virtual reality. Current 3D generation methods utilizing NeRF or 3D-GS as representations for 3D objects, generate a Lambertian object under fixed lighting and lack separated modelings for material and global illumination. As a result, the generated assets are unsuitable for relighting under varying lighting conditions, limiting their applicability in downstream tasks. To address this challenge, we propose a novel relightable 3D object generative framework that automates the creation of 3D car assets, enabling the swift and accurate reconstruction of a vehicle's geometry, texture, and material properties from a single input image. Our approach begins with introducing a large-scale synthetic car dataset comprising over 1,000 high-precision 3D vehicle models. We represent 3D objects using global illumination and relightable 3D Gaussian primitives integrating with BRDF parameters. Building on this representation, we introduce a feed-forward model that takes images as input and outputs both relightable 3D Gaussians and global illumination parameters. Experimental results demonstrate that our method produces photorealistic 3D car assets that can be seamlessly integrated into road scenes with different illuminations, which offers substantial practical benefits for industrial applications.

Figures (6)

• Figure 1: Our method generates high-quality 3D cars and corresponding material properties from the single image, which allows for realistic relighting under different lighting conditions.
• Figure 2: The pipeline for constructing Carverse dataset.
• Figure 3: Overall architecture of RGM. We first input a single image into the multi-view generation model to obtain multi-view consistent images of the car. These images and camera embeddings are then fed into the relightable 3D-GS generative model, which produces both global illumination parameters and relightable Gaussian representations. Through a physically-based rendering layer, we can obtain the material properties of the car and relight the car with a new illumination.
• Figure 4: Qualitative results of generated novel views, compared with TripoSR tochilkin2024triposr,InstantMesh xu2024instantmesh and Real3D jiang2024real3d.
• Figure 5: Visualization of gaussian points with or without relightable gaussians.