Hybrid Fourier Score Distillation for Efficient One Image to 3D Object Generation

TL;DR

A 2D-3D hybrid Fourier score distillation objective function, called hy-FSD, that optimizes 3D Gaussians using 3D priors in the spatial domain to ensure geometric consistency, while exploiting 2D priors in the frequency domain through the Fourier transform for better visual quality.

Abstract

Single image-to-3D generation is pivotal for crafting controllable 3D assets. Given its under-constrained nature, we attempt to leverage 3D geometric priors from a novel view diffusion model and 2D appearance priors from an image generation model to guide the optimization process. We note that there is a disparity between the generation priors of these two diffusion models, leading to their different appearance outputs. Specifically, image generation models tend to deliver more detailed visuals, whereas novel view models produce consistent yet over-smooth results across different views. Directly combining them leads to suboptimal effects due to their appearance conflicts. Hence, we propose a 2D-3D hybrid Fourier Score Distillation objective function, hy-FSD. It optimizes 3D Gaussians using 3D priors in spatial domain to ensure geometric consistency, while exploiting 2D priors in the frequency domain through Fourier transform for better visual quality. hy-FSD can be integrated into existing 3D generation methods and produce significant performance gains. With this technique, we further develop an image-to-3D generation pipeline to create high-quality 3D objects within one minute, named Fourier123. Extensive experiments demonstrate that Fourier123 excels in efficient generation with rapid convergence speed and visually-friendly generation results.

Figures (14)

• Figure 1: Fourier123 aims at increasing the generation quality of image-to-3D task. We are able to generate a high-quality 3D object that is highly consistent with the input image within one minute.
• Figure 2: Frequency analysis of Stable Diffusion (SD) and Zero-1-to-3 (Zero123). Discrete Fourier Transform (DFT) converts the results of SD ($S_{2D}$) and Zero123 ($S_{3D}$) to frequency domain and here we visualize their amplitude components. In the upper row, $S_{2D}$ exhibits high visual quality but distorts content structure. In the lower row, $S_{3D}$ matches the input but is over-smooth. Their frequency amplitudes, $F_{3D}$ and $F_{2D}$, are also different. We train with $S_{3D}$ for its fidelity, and $F_{2D}$ for finer details. More details can be found in Fig. \ref{['fig:workflow']}.
• Figure 3: The workflow of Fourier123. We first use $\mathcal{F}(\cdot)$ to initialize 3D Gaussian $\boldsymbol{\phi}$. $\mathcal{F}(\cdot)$ can be sphere initialization or large reconstruction model. Then, Zero123 0123 is used to supervise geometry in the spatial domain, while SD sd supervises appearance in the frequency domain. The whole generation process takes less than one minute.
• Figure 4: Visual results of ablation study on hy-FSD. We input a single image and a prompt, where the prompt is generated by ChatGPT based on the image. One can see that settings that use 2D-SDS to supervise in spatial domain all exhibit content distortion. Our hy-FSD achieves the best results.
• Figure 5: Visual comparison. Input images are given on the left and runtime is listed below. For clear comparison, we omit LGM and InstantMesh here and the full version can be found in Sec. \ref{['sec:comple_subcompare']}.