EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion

TL;DR

EpiDiff tackles the challenge of fast, consistent multiview synthesis from a single image by introducing a localized Epipolar-constrained Attention Block that plugs into a frozen diffusion backbone. The method leverages Near-Views Cross-Attention and Ray Self-Attention, guided by epipolar geometry and Light Field Network–style ray encoding, to model 3D consistency without heavy global 3D representations. Key contributions include a training scheme that preserves the base diffusion model, a two-component attention module that enables cross-view interaction, and strong empirical results showing 16 views in 12 seconds with superior multiview quality and reconstruction metrics. The approach yields more diverse view distributions, improving 3D reconstruction while maintaining compatibility with multiple base diffusion models, making it practical for fast, high-fidelity novel view synthesis from a single image.

Abstract

Generating multiview images from a single view facilitates the rapid generation of a 3D mesh conditioned on a single image. Recent methods that introduce 3D global representation into diffusion models have shown the potential to generate consistent multiviews, but they have reduced generation speed and face challenges in maintaining generalizability and quality. To address this issue, we propose EpiDiff, a localized interactive multiview diffusion model. At the core of the proposed approach is to insert a lightweight epipolar attention block into the frozen diffusion model, leveraging epipolar constraints to enable cross-view interaction among feature maps of neighboring views. The newly initialized 3D modeling module preserves the original feature distribution of the diffusion model, exhibiting compatibility with a variety of base diffusion models. Experiments show that EpiDiff generates 16 multiview images in just 12 seconds, and it surpasses previous methods in quality evaluation metrics, including PSNR, SSIM and LPIPS. Additionally, EpiDiff can generate a more diverse distribution of views, improving the reconstruction quality from generated multiviews. Please see our project page at https://huanngzh.github.io/EpiDiff/.

Figures (8)

• Figure 1: EpiDiff is able to efficiently generate multi-view consistent and high-quality images from a single view input. Instead of limited to fixed viewpoints, EpiDiff can generate relatively arbitrary multi-views. EpiDiff is lightweight and only takes 12 seconds to generate 16 multi-view images. The generated multiview images can be used to recover 3D shapes by neural reconstruction methods wang2021neusinstant-nsr-pl.
• Figure 2: Pipeline of EpiDiff. Based on a base NVS model (e.g., Zero123 zero123), our method designs a module for modeling 3D consistency, which is inserted into the mid-sample and up-sample stages of the UNet. We use attention networks to construct the module, aimed at learning generalized epipolar geometry, termed Epipolar-constrained Attention Block (ECA Block). During training, only parameters of the ECA Block are updated, thereby preserving the feature space of the base model and encouraging the module to extract 3D priors.
• Figure 3: Illustration of our Epipolar-constrained Attention Block (ECA Block). Latent features of multiview images are associated in 3D space through two key attention blocks. The Near-Views Cross-Attn initially aggregates features from nearby views onto the target view's ray points, guided by epipolar geometry. Subsequently, Ray Self-Attn models depth information to fuse ray features into 2D feature maps. The ECA Block facilitates multiview interaction by effectively harnessing spatial geometry.
• Figure 4: Qualitative comparison with Zero123 zero123 and SyncDreamer syncdreamer under the elevation $30^{\circ}$ setting.
• Figure 5: Qualitative comparison with Zero123 zero123 and SyncDreamer syncdreamer under the uniform elevation setting. We present the generated results of each method when the elevation is $-10^{\circ}$, $10^{\circ}$ and $30^{\circ}$.