Geometric Consistency Refinement for Single Image Novel View Synthesis via Test-Time Adaptation of Diffusion Models

TL;DR

This work tackles geometric misalignment in diffusion-based single-image novel view synthesis by introducing GC-Ref, a training-free test-time refinement that optimizes the diffusion starting noise to satisfy epipolar constraints and photometric consistency with the reference view. The method uses a differentiable RoMa-based matcher to obtain dense correspondences and a fundamental matrix $F$ derived from the target pose to define a geometric loss, which is backpropagated through the diffusion process during DDIM sampling. Applied to state-of-the-art models on the MegaScenes dataset, GC-Ref yields improved pose accuracy and closer epipolar alignment while maintaining or enhancing image quality, all without additional training. This approach provides a practical, data-efficient pathway to enforce geometric fidelity in diffusion-driven NVS, with potential impact on robotics, AR, and multi-view applications where pose precision is critical.

Abstract

Diffusion models for single image novel view synthesis (NVS) can generate highly realistic and plausible images, but they are limited in the geometric consistency to the given relative poses. The generated images often show significant errors with respect to the epipolar constraints that should be fulfilled, as given by the target pose. In this paper we address this issue by proposing a methodology to improve the geometric correctness of images generated by a diffusion model for single image NVS. We formulate a loss function based on image matching and epipolar constraints, and optimize the starting noise in a diffusion sampling process such that the generated image should both be a realistic image and fulfill geometric constraints derived from the given target pose. Our method does not require training data or fine-tuning of the diffusion models, and we show that we can apply it to multiple state-of-the-art models for single image NVS. The method is evaluated on the MegaScenes dataset and we show that geometric consistency is improved compared to the baseline models while retaining the quality of the generated images.

Figures (7)

• Figure 1: The single image novel view synthesis task is to, given a reference image and a relative pose, generate an image of the scene from the target pose. The estimated pose for an image generated by the diffusion based method ZeroNVS is shown in red and our refined estimate is depicted in green. The reference pose is shown in blue and the target pose in black. As can be seen, the estimated relative poses from the image generated by the diffusion model can differ significantly from the target pose. Our method refines such images to better align with the target pose.
• Figure 2: Our method for geometric consistency refinement (GC-Ref) modifies images such that corresponding points in the reference image and the generated image lie close to their corresponding epipolar lines. We show an example of a reference image with a warping to the target pose, obtained via monocular depth estimation, that the generated image should align with. If we consider matching points between the reference images and the generated images we see that after our refinement the points lie closer to their epipolar lines. This is also shown in the histograms where we show the distributions of the distances between matching points and their corresponding epipolar lines before and after our refinement.
• Figure 3: Our method for geometric consistency refinement aims to iteratively refine an image $x_0$ generated by diffusion model $\epsilon_\theta$ for single image view synthesis to better fulfill geometric constraints. Our method is based on the fact that if the images are geometrically consistent given the target pose then all matching points between the reference image $I_{ref}$ and the generated image $x_0$ should lie on the corresponding epipolar lines. We explicitly optimize this criteria by computing matching points between the reference image and generated image via a differentiable matcher and then use the epipolar distances as a loss function $\mathcal{L}$ to optimize the starting noise $z_T$ of the diffusion process using the gradient $\nabla_{z_T} \mathcal{L}$.
• Figure 4: Given the reference image and generated image we estimate the relative poses and compare it with the target poses. We show how the pose errors vary with the magnitude of the rotation of the target pose. The rotation errors increase the more the target pose is rotated from the reference pose, while the translation errors are largest for small camera motions. For both measures we see that with our geometric consistency refinement (GC-Ref) the errors are reduced compared to the baseline methods.
• Figure 5: Our refinement leads to reduced epipolar distances and to improved pose accuracy. We see in the histogram that the distances to the epipolar lines decrease after our refinement and also in the 3D plot that the estimated camera pose (green) is closer to the target pose (black) than the original generated image (red). In this specific case the translation error decreased from $2.8^{\circ}$ to $0.8^{\circ}$ and the rotation error from $1.8^{\circ}$ to $0.8^{\circ}$.