iLRM: An Iterative Large 3D Reconstruction Model

TL;DR

iLRM addresses scalability bottlenecks in feed-forward 3D reconstruction by decoupling Gaussian generation from input views and applying an iterative refinement process with a two-stage attention mechanism. It learns viewpoint embeddings via Plücker ray representations, cross-attends with per-view image tokens, then self-attends across views to update a compact 3D Gaussian representation which is decoded into parameters $({\\mu}_k, {\\alpha}_k, {\\Sigma}_k, {\\ c}_k)$. The model achieves higher reconstruction fidelity with lower computational cost than prior feed-forward methods on RealEstate10K and DL3DV, and even outperforms some optimization-based methods in wide-coverage regimes, with inference times around 0.5 seconds. These results show that iterative, view-informed refinement with decoupled representations can combine fast inference and 3D consistency for scalable generalizable 3D reconstruction.

Abstract

Feed-forward 3D modeling has emerged as a promising approach for rapid and high-quality 3D reconstruction. In particular, directly generating explicit 3D representations, such as 3D Gaussian splatting, has attracted significant attention due to its fast and high-quality rendering, as well as numerous applications. However, many state-of-the-art methods, primarily based on transformer architectures, suffer from severe scalability issues because they rely on full attention across image tokens from multiple input views, resulting in prohibitive computational costs as the number of views or image resolution increases. Toward a scalable and efficient feed-forward 3D reconstruction, we introduce an iterative Large 3D Reconstruction Model (iLRM) that generates 3D Gaussian representations through an iterative refinement mechanism, guided by three core principles: (1) decoupling the scene representation from input-view images to enable compact 3D representations; (2) decomposing fully-attentional multi-view interactions into a two-stage attention scheme to reduce computational costs; and (3) injecting high-resolution information at every layer to achieve high-fidelity reconstruction. Experimental results on widely used datasets, such as RE10K and DL3DV, demonstrate that iLRM outperforms existing methods in both reconstruction quality and speed.

Figures (14)

• Figure 1: The overall architecture and qualitative results of the proposed iLRM.
• Figure 2: The proposed scalable architectural designs by decoupling viewpoint and image tokens, and modeling the global interactions via cross- and self-attentions ($N$: # views, $h =H/p, w = W/p$).
• Figure 3: Network architecture.
• Figure 4: Various mini-batch cross-attention schemes.
• Figure 5: Qualitative comparison of novel view synthesis on the RE10K dataset (256$\times$256).