Fixing the Perspective: A Critical Examination of Zero-1-to-3

TL;DR

This work conducts a thorough investigation of Zero-1-to-3's cross-attention mechanism within the Spatial Transformer of the diffusion 2D-conditional UNet and proposes two significant improvements: a corrected implementation that enables effective utilization of the cross-attention mechanism, and an enhanced architecture that can leverage multiple conditional views simultaneously.

Abstract

Novel view synthesis is a fundamental challenge in image-to-3D generation, requiring the generation of target view images from a set of conditioning images and their relative poses. While recent approaches like Zero-1-to-3 have demonstrated promising results using conditional latent diffusion models, they face significant challenges in generating consistent and accurate novel views, particularly when handling multiple conditioning images. In this work, we conduct a thorough investigation of Zero-1-to-3's cross-attention mechanism within the Spatial Transformer of the diffusion 2D-conditional UNet. Our analysis reveals a critical discrepancy between Zero-1-to-3's theoretical framework and its implementation, specifically in the processing of image-conditional context. We propose two significant improvements: (1) a corrected implementation that enables effective utilization of the cross-attention mechanism, and (2) an enhanced architecture that can leverage multiple conditional views simultaneously. Our theoretical analysis and preliminary results suggest potential improvements in novel view synthesis consistency and accuracy.

Figures (6)

• Figure 1: Architectural overview of multi-view conditional generation. The model processes multiple input views and their corresponding camera poses to generate novel viewpoints.
• Figure 2: Cross-attention mechanism within a single U-Net layer during the diffusion process of Zero-1-to-3. The attention weights exhibit unexpected behavior due to architectural constraints.
• Figure 3: Example of back-view generation artifacts from RealFusion melaskyriazi2023realfusion, demonstrating the limitations of single-view conditioning.
• Figure 4: Multi-view architecture overview. Multiple input views and their camera poses are processed through parallel encoding paths before being combined for novel view generation.
• Figure 5: Architecture for our proposed fix to Zero-1-to-3 with separate positional information embedding. Image courtesy of Deepsense.ai