MEt3R: Measuring Multi-View Consistency in Generated Images

TL;DR

MEt3R introduces a pose-free, image-pair metric for evaluating multi-view consistency in generated scenes by warping views through dense 3D reconstructions and comparing projected features in a shared view. It combines DUSt3R-based reconstruction with upsampled DINO features (via FeatUp) to compute a robust similarity score and defines $MEt3R$ as a symmetric, continuous measure in $[0,2]$, independent of ground-truth camera poses. The authors also present MV-LDM, an open-source multi-view latent diffusion model trained on RealEstate10k with an anchored generation strategy to improve 3D consistency. Across RealEstate10K sequences, MEt3R distinguishes fine-grained consistency differences between baselines and shows that higher 3D consistency does not necessarily compromise image quality, underscoring its value for benchmarking future 3D-aware generative models. The pose-free and differentiable nature of MEt3R makes it well-suited for evaluating both multi-view generation and video synthesis in the absence of explicit camera information, with broad practical impact for 3D-aware generation research.

Abstract

We introduce MEt3R, a metric for multi-view consistency in generated images. Large-scale generative models for multi-view image generation are rapidly advancing the field of 3D inference from sparse observations. However, due to the nature of generative modeling, traditional reconstruction metrics are not suitable to measure the quality of generated outputs and metrics that are independent of the sampling procedure are desperately needed. In this work, we specifically address the aspect of consistency between generated multi-view images, which can be evaluated independently of the specific scene. Our approach uses DUSt3R to obtain dense 3D reconstructions from image pairs in a feed-forward manner, which are used to warp image contents from one view into the other. Then, feature maps of these images are compared to obtain a similarity score that is invariant to view-dependent effects. Using MEt3R, we evaluate the consistency of a large set of previous methods for novel view and video generation, including our open, multi-view latent diffusion model.

Figures (19)

• Figure 1: We introduce MEt3R, a metric for multi-view consistency between pairs of generated images, which is independent of image quality, image content, and does not require camera poses. Left: generated images from different generative models, conditioned on the first frame, with MEt3R score map indicating levels of inconsistencies between consecutive images $i$ and $i+1$. Right: pair-wise consistency scores, evaluated for consecutive frames in a sliding window, averaged over multiple sequences. The pattern in MV-LDM's consistency clearly shows artifacts from using anchor frames that are generated first, highlighting the high signal-to-noise ratio of MEt3R.
• Figure 2: Existing metrics. A comparison between MEt3R and TSED Yu2023PhotoconsistentNVS scores obtained from individual image pairs generated by GenWarp seo2024genwarp. TSED misses obvious, partial multi-view inconsistencies and is biased to small violations of epipolar geometry. In contrast, MEt3R correctly captures clear 3D inconsistencies and is robust to insignificant artifacts almost invisible to the human eye.
• Figure 3: Method overview. Our metric evaluates the consistency between images $\mathbf{I}_1$ and $\mathbf{I}_2$. Given such a pair, we apply DUSt3R to obtain dense 3D point maps $\mathbf{X}_1$ and $\mathbf{X}_2$. These point maps are used to project upscaled DINO features $\mathbf{F}_1$, $\mathbf{F}_2$ into the coordinate frame of $\mathbf{I}_1$, via unprojecting and rendering. We compare the resulting feature maps $\hat{\mathbf{F}}_1$ and $\hat{\mathbf{F}}_2$ in pixel space to obtain similarity $S(\mathbf{I}_1,\mathbf{I}_2)$.
• Figure 4: Metric comparison. We compare MEt3R against TSED, SED, and FVD by computing average per-frame (/-segment for FVD) scores over a large number of generated sequences. MEt3R is able to capture nuanced differences in consistency of DFM, MV-LDM, and real videos, while TSED rates them all very similar. Unlike MEt3R, SED does not capture increasing inconsistency for PhotoNVS and DFM. MEt3R is able to capture the influence of anchor views in MV-LDM (c.f. Sec. \ref{['sec:mv_ldm']} and appendix Sec. \ref{['sec:mv_ldm_details']}) as structured high-frequency patterns. For MEt3R, the standard deviation gradually increases, starting from a small value, which is expected behavior due to conditioning on the first frame and is not the case for the other metrics.
• Figure 5: Qualitative comparison of generated novel views. We compare generated views of the multi-view generation method for the same conditioning view. We can extract certain characteristics: DFM is almost perfectly consistent but has lower image quality. PhotoNVS and MV-LDM are reasonably consistent on a structural scale but fail to produce consistent details. GenWarp fails to keep the structural consistency over the sequence while producing high-quality images. These observations are confirmed by MEt3R in Tab. \ref{['tab:metrics']} and Fig. \ref{['fig:metric_comparison']}.