Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models

TL;DR

Hi3D addresses the challenge of producing high-resolution, multi-view-consistent 3D reconstructions from a single image by reformulating image-to-3D as 3D-aware orbital video generation. It employs a two-stage video diffusion pipeline: Stage-1 remolds a video diffusion model with camera pose cues to generate low-resolution orbit views, and Stage-2 uses a depth-conditioned 3D-aware refiner to reach high-resolution views, which are augmented via 3D Gaussian Splatting before SDF-based mesh extraction. The approach yields state-of-the-art results in novel view synthesis and single-view reconstruction on high-resolution data, demonstrating superior texture detail and geometry consistency. The method offers a scalable path to high-fidelity 3D content for VR and film production, with source code and data available to the community.

Abstract

Despite having tremendous progress in image-to-3D generation, existing methods still struggle to produce multi-view consistent images with high-resolution textures in detail, especially in the paradigm of 2D diffusion that lacks 3D awareness. In this work, we present High-resolution Image-to-3D model (Hi3D), a new video diffusion based paradigm that redefines a single image to multi-view images as 3D-aware sequential image generation (i.e., orbital video generation). This methodology delves into the underlying temporal consistency knowledge in video diffusion model that generalizes well to geometry consistency across multiple views in 3D generation. Technically, Hi3D first empowers the pre-trained video diffusion model with 3D-aware prior (camera pose condition), yielding multi-view images with low-resolution texture details. A 3D-aware video-to-video refiner is learnt to further scale up the multi-view images with high-resolution texture details. Such high-resolution multi-view images are further augmented with novel views through 3D Gaussian Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D reconstruction. Extensive experiments on both novel view synthesis and single view reconstruction demonstrate that our Hi3D manages to produce superior multi-view consistency images with highly-detailed textures. Source code and data are available at \url{https://github.com/yanghb22-fdu/Hi3D-Official}.

Figures (6)

• Figure 1: An overview of our proposed Hi3D. Our Hi3D fully exploits the capabilities of large-scale pre-trained video diffusion models to effectively trigger high-resolution image-to-3D generation. Specifically, in the first stage of basic multi-view generation, Hi3D remoulds video diffusion model with additional camera pose condition, aiming to transform single-view image into low-resolution 3D-aware sequential images. Next, in the second stage of 3D-aware multi-view refinement, we feed this low-resolution orbit video into 3D-aware video-to-video refiner with additional depth condition, leading to high-resolution orbit video with highly detailed texture. Finally, we augment the resultant multi-view images with more novel views through 3D Gaussian Splatting and employ SDF-based reconstruction to extract high-quality 3D meshes.
• Figure 2: Qualitative comparisons with Stable-Zero123 Stable-zero123, SyncDreamer liu2023syncdreamer and EpiDiff huang2023epidiff on novel view synthesis task. Our Hi3D generates high-resolution multi-view images with remarkable consistent details.
• Figure 3: Qualitative comparison of 3D meshes generated by various methods on single view reconstruction task.
• Figure 4: Examples of using Hi3D for text-to-3D generation.
• Figure 5: Diverse and creative results of our Hi3D with different seeds.