WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space

TL;DR

WildFusion tackles 3D-aware image synthesis on unposed, in-the-wild data by modeling instances in view space and learning a 3D-aware latent representation via a two-stage process: a 3D-aware autoencoder that enables novel-view synthesis and compression, and a latent diffusion model trained on the autoencoder’s latent space. Monocular depth cues are incorporated to guide geometry without multiview supervision, and a KL-regularized latent space enables efficient diffusion-based generation with 3D consistency. The approach achieves state-of-the-art performance on non-aligned datasets like SDIP and ImageNet, outperforms GAN-based 3D-aware methods in terms of geometry and diversity, and enables 3D-aware interpolation and resampling. This work paves the way for scalable 3D content creation from unposed images, broadening applicability to diverse real-world datasets and workflows.

Abstract

Modern learning-based approaches to 3D-aware image synthesis achieve high photorealism and 3D-consistent viewpoint changes for the generated images. Existing approaches represent instances in a shared canonical space. However, for in-the-wild datasets a shared canonical system can be difficult to define or might not even exist. In this work, we instead model instances in view space, alleviating the need for posed images and learned camera distributions. We find that in this setting, existing GAN-based methods are prone to generating flat geometry and struggle with distribution coverage. We hence propose WildFusion, a new approach to 3D-aware image synthesis based on latent diffusion models (LDMs). We first train an autoencoder that infers a compressed latent representation, which additionally captures the images' underlying 3D structure and enables not only reconstruction but also novel view synthesis. To learn a faithful 3D representation, we leverage cues from monocular depth prediction. Then, we train a diffusion model in the 3D-aware latent space, thereby enabling synthesis of high-quality 3D-consistent image samples, outperforming recent state-of-the-art GAN-based methods. Importantly, our 3D-aware LDM is trained without any direct supervision from multiview images or 3D geometry and does not require posed images or learned pose or camera distributions. It directly learns a 3D representation without relying on canonical camera coordinates. This opens up promising research avenues for scalable 3D-aware image synthesis and 3D content creation from in-the-wild image data. See https://katjaschwarz.github.io/wildfusion for videos of our 3D results.

Figures (20)

• Figure 1: WildFusion:Left: Input images, novel views and geometry from first-stage autoencoder. Right: Novel samples and geometry from our second-stage latent diffusion model and 3DGP skorokhodov20233dgp for the ImageNet classes "macaw" (top), "king penguin" (middle), "kimono" (bottom). Included videos for more results.
• Figure 2: Sample Diversity: Generated samples on ImageNet. Rows indicate class; columns show uncurated random samples. While WildFusion generates diverse samples due to its diffusion model-based framework (left), the GAN-based 3DGP skorokhodov20233dgp has very low intra-class diversity (mode collapse, right).
• Figure 3: WildFusion Overview: In the first stage, we train an autoencoder for both compression and novel-view synthesis. A Feature Pyramid Network (FPN) Lin2017CVPRb encodes a given unposed image $\mathbf{I}$ into an 3D-aware latent representation $\mathbf{Z}$, constructed as a 2D feature grid. A combination of transformer blocks and a CNN then decode $\mathbf{Z}$ into a triplane representation, which is rendered from both the input view $\mathbf{P}_0$ and a novel view $\mathbf{P}_{nv}$. As we model instances in view space, $\mathbf{P}_0$ is a fixed, pre-defined camera pose. The input view is supervised with reconstruction losses. Adversarial training provides supervision for novel views. In the second stage, a latent diffusion model is trained on the learned latent space to obtain a 3D-aware generative model.
• Figure 4: Baseline comparisons for novel view synthesis on images unseen during training. Shown are the input image and two novel views per method. Viewpoints across methods are the same. Included video for more results.
• Figure 5: Generated 3D-aware image samples and geometry by WildFusion. Included videos for more results.