RIGI: Rectifying Image-to-3D Generation Inconsistency via Uncertainty-aware Learning

TL;DR

Addresses the challenge of cross-view inconsistencies in image-to-3D generation from a single image. Introduces uncertainty-aware learning using a dual Gaussian ensemble within 3D Gaussian Splatting (3DGS) to estimate and regularize against view-induced artifacts, guided by multi-view pseudo-labels from SV3D. The approach combines adaptive pixel-wise loss weighting with LPIPS and D-SSIM terms, and reinforces stability through optimization strategies like PAG and progressive sampling. Experiments on the GSO benchmark show improved 3D quality and strong user preference, demonstrating practical gains for robust, texture-rich 3D asset generation from a single image.

Abstract

Given a single image of a target object, image-to-3D generation aims to reconstruct its texture and geometric shape. Recent methods often utilize intermediate media, such as multi-view images or videos, to bridge the gap between input image and the 3D target, thereby guiding the generation of both shape and texture. However, inconsistencies in the generated multi-view snapshots frequently introduce noise and artifacts along object boundaries, undermining the 3D reconstruction process. To address this challenge, we leverage 3D Gaussian Splatting (3DGS) for 3D reconstruction, and explicitly integrate uncertainty-aware learning into the reconstruction process. By capturing the stochasticity between two Gaussian models, we estimate an uncertainty map, which is subsequently used for uncertainty-aware regularization to rectify the impact of inconsistencies. Specifically, we optimize both Gaussian models simultaneously, calculating the uncertainty map by evaluating the discrepancies between rendered images from identical viewpoints. Based on the uncertainty map, we apply adaptive pixel-wise loss weighting to regularize the models, reducing reconstruction intensity in high-uncertainty regions. This approach dynamically detects and mitigates conflicts in multi-view labels, leading to smoother results and effectively reducing artifacts. Extensive experiments show the effectiveness of our method in improving 3D generation quality by reducing inconsistencies and artifacts.

Figures (8)

• Figure 1: Given an input image, our method mitigates the impact of inconsistencies between generated dense frames on 3D asset optimization, reducing edge artifacts and floats while producing visually impressive 3D objects. We sample six rendered images uniformly across an azimuth range of 0 to 360°, with elevations following a sine function with a 30° amplitude, effectively capturing front, top, back, and bottom perspectives, which are crucial to real-world applications, yet often overlooked by most existing methods.
• Figure 2: Prevailing Image-to-3D methods typically adopt the synthesized video as an intermediate representation to guide the 3D object generation. However, the frame-to-frame inconsistencies can lead to incorrect geometry and artifacts in the 3D assets. In this example, red bounding boxes highlight extra toy arms and clock legs, which represent common failures in the generation process.
• Figure 3: Overview of our pipeline. Firstly, we use SV3D sv3d to generate multiple videos with a wide range of viewpoints, which serve as pseudo-labels for 3D asset optimization. Next, we introduce uncertainty-aware learning, estimating an uncertainty map by leveraging the stochasticity of two simultaneously optimized Gaussian models. Finally, we apply uncertainty-aware regularization to mitigate the impact of inconsistencies in the generated pseudo-labels, resulting in high-quality and visually impressive 3D assets.
• Figure 4: Visual Comparison. Here we compare competitive image-to-3D methods, including TriplaneGaussian triplanegaussian, LGM lgm, Dreamgaussian dreamgaussian, V3D v3d and Hi3D hi3d. We achieve visually impressive results, with high-quality geometric and texture details even from top and bottom perspectives.
• Figure 5: Ablation analysis on the impact of progressive sampling and uncertainty-aware learning. These techniques effectively mitigate artifacts, floats, and geometric deformations in inconsistent regions, resulting in visually enhanced 3D assets.