WildCAT3D: Appearance-Aware Multi-View Diffusion in the Wild

TL;DR

WildCAT3D tackles the challenge of scene-level novel-view synthesis from diverse in-the-wild images by introducing appearance-aware multi-view diffusion and warp conditioning. Building on CAT3D, it adds a lightweight appearance encoder and depth-guided warps to decouple appearance from content and improve viewpoint consistency, enabling appearance-controlled generation and transfer. The method demonstrates state-of-the-art performance on single-view NVS benchmarks, using fewer data sources than prior work and enabling scalable training from permissively licensed web data. Its ability to interpolate appearances and control look-and-feel through external cues suggests a significant step toward web-scale, controllable NVS for full scenes and related applications.

Abstract

Despite recent advances in sparse novel view synthesis (NVS) applied to object-centric scenes, scene-level NVS remains a challenge. A central issue is the lack of available clean multi-view training data, beyond manually curated datasets with limited diversity, camera variation, or licensing issues. On the other hand, an abundance of diverse and permissively-licensed data exists in the wild, consisting of scenes with varying appearances (illuminations, transient occlusions, etc.) from sources such as tourist photos. To this end, we present WildCAT3D, a framework for generating novel views of scenes learned from diverse 2D scene image data captured in the wild. We unlock training on these data sources by explicitly modeling global appearance conditions in images, extending the state-of-the-art multi-view diffusion paradigm to learn from scene views of varying appearances. Our trained model generalizes to new scenes at inference time, enabling the generation of multiple consistent novel views. WildCAT3D provides state-of-the-art results on single-view NVS in object- and scene-level settings, while training on strictly less data sources than prior methods. Additionally, it enables novel applications by providing global appearance control during generation.

Figures (11)

• Figure 1: WildCAT3D. (top) We use large image collections captured in the wild to train our feed-forward novel-view synthesis model. (middle) At inference time, WildCAT3D can generate full scene-level novel views from a single image of a new (never-before encountered) scene. (bottom) It can also be used to control the appearance of the generated views, e.g. via a text prompt.
• Figure 2: Overview. WildCAT3D learns to synthesize novel views by denoising target views of inconsistent appearances from a source view. Given a batch of source (blue) and target (red) views, we first compute camera embeddings and VAE latents. The latter are then fed to an encoder computing a small appearance vector copied across spatial locations, allowing the model to "peek" at appearance conditions. Finally, for target views, we compute additional warping embeddings using the VAE applied to warped source images calculated from an estimated depth map. These signals are channel-wise concatenated and fed to the diffusion model. During training (depicted above), noise is added to target view latents. During inference, target view latents are replaced by random noise while their appearance channels are copied from the source view branch.
• Figure 3: Qualitative comparison on MegaScenes with novel trajectories. Using single images as input (left), we show results for WildCAT3D and MegaScenes NVS model (MS NVS) on scenes unseen during training, conditioned on a continuous camera trajectory. We see that our model significantly outperforms prior SOTA at generating consistent and high-quality sequences from single views. We encourage the reader to check our video results in our supmat to further assess the quality gap.
• Figure 4: Application: appearance-controlled generation. Starting from a source view (left) and an additional image with a specific appearance (middle), our model is able to synthesize novel views that are not only consistent with the source view content and the desired viewpoints, but also consistent with the appearance style of the additional image (right). We perform text-guidance by concatenating our model with a text-to-image retrieval model. $^*$Retrieved with text prompts "sunset" and "a spring day with a clear blue sky" respectively.
• Figure 5: Application: in-the-wild interpolation. When fine-tuned with two observed views, WildCAT3D can interpolate between scene views with differing appearances. Injecting the appearance embedding of either the start or the end pose yields generated views with consistent appearances.