MVInverse: Feed-forward Multi-view Inverse Rendering in Seconds

TL;DR

<MVInverse> addresses multi-view inverse rendering by delivering a fast, feed-forward framework that predicts scene-intrinsic maps for all views in a sequence. It leverages an alternating global/frame-attention transformer to fuse cross-view consistency with intra-view details, augmented by a multi-resolution encoder and a suite of intrinsic heads. A two-stage training regimen—synthetic pretraining plus consistency finetuning on real videos—enhances real-world generalization and mitigates flickering artifacts. The approach achieves state-of-the-art multi-view consistency and normal estimation while enabling practical applications like relighting and view-consistent material editing. This work significantly speeds up multi-view material understanding, enabling scalable, data-driven PBR pipelines for real-world imagery and video.

Abstract

Multi-view inverse rendering aims to recover geometry, materials, and illumination consistently across multiple viewpoints. When applied to multi-view images, existing single-view approaches often ignore cross-view relationships, leading to inconsistent results. In contrast, multi-view optimization methods rely on slow differentiable rendering and per-scene refinement, making them computationally expensive and hard to scale. To address these limitations, we introduce a feed-forward multi-view inverse rendering framework that directly predicts spatially varying albedo, metallic, roughness, diffuse shading, and surface normals from sequences of RGB images. By alternating attention across views, our model captures both intra-view long-range lighting interactions and inter-view material consistency, enabling coherent scene-level reasoning within a single forward pass. Due to the scarcity of real-world training data, models trained on existing synthetic datasets often struggle to generalize to real-world scenes. To overcome this limitation, we propose a consistency-based finetuning strategy that leverages unlabeled real-world videos to enhance both multi-view coherence and robustness under in-the-wild conditions. Extensive experiments on benchmark datasets demonstrate that our method achieves state-of-the-art performance in terms of multi-view consistency, material and normal estimation quality, and generalization to real-world imagery.

Figures (17)

• Figure 1: We present MVInverse, a feed-forward multi-view inverse rendering framework that jointly recovers consistent geometry and material properties from input images or videos. With the recovered properties, realistic relighting under novel illumination can be achieved within seconds.
• Figure 2: Alternating Attention. Given the red and blue query patches in the first view, we visualize the corresponding attention heatmaps in the second view (bottom row) to illustrate the effectiveness of our model design. Red: the model captures long-range lighting interactions across spatially distant regions. Blue: the model maintains cross-view consistency by correctly associating corresponding surface regions under viewpoint changes.
• Figure 3: Framework of MVInverse. Given an input image sequence, our framework first encodes each frame into patch tokens via DINOv2 oquab2023dinov2, where alternating global–frame attention enables cross-view feature aggregation. Meanwhile, a frame-wise ResNeXt xie2017aggregated encoder provides multi-resolution convolutional features to preserve fine spatial details. The two feature streams are fused in a DPT-style prediction head to produce pixel-aligned intrinsic maps, including albedo, metallic, roughness, normal, and shading. The diffuse image is obtained as the product of albedo and diffuse shading.
• Figure 4: Consistency finetuning on real videos. Frames are fed into the finetuned model to produce predictions. Optical flow warps the prediction of frame $t+1$ to frame $t$, and a consistency loss enforces temporal stability. An anchor loss on frame $0$, using a pretrained model's prediction as reference, prevents solution collapse.
• Figure 5: Single-view Albedo Comparison. In the top row, green circles highlight regions where lighting is not removed; blue circles highlights regions where shadow still exists; orange circles demonstrate an unnatural red patch inside the leaves. In the bottom row, red and blue regions indicate areas where small shadows are not removed; green regions highlight shadows cast by the pendant light on the cabinet that are not recognized by other models. In contrast, our model handles these cases well.