LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields

TL;DR

LIRM tackles the challenge of recovering geometry, materials, and lighting from sparse multi-view images by introducing a fast, progressive transformer-based framework. It combines a progressive update mechanism, a hexaplane neural SDF representation for detailed shape and materials, and neural directional encoding to model view-dependent radiance, enabling relightable 3D content. The approach is trained on a large synthetic dataset with a coarse-to-fine scheme and demonstrates competitive geometry and relighting accuracy while significantly reducing inference time relative to optimization-based inverse rendering. The work enables practical 3D content creation for graphics pipelines and real-world applications by delivering fast, relightable reconstructions from few views that can be edited and rendered under novel lighting conditions.

Abstract

We present Large Inverse Rendering Model (LIRM), a transformer architecture that jointly reconstructs high-quality shape, materials, and radiance fields with view-dependent effects in less than a second. Our model builds upon the recent Large Reconstruction Models (LRMs) that achieve state-of-the-art sparse-view reconstruction quality. However, existing LRMs struggle to reconstruct unseen parts accurately and cannot recover glossy appearance or generate relightable 3D contents that can be consumed by standard Graphics engines. To address these limitations, we make three key technical contributions to build a more practical multi-view 3D reconstruction framework. First, we introduce an update model that allows us to progressively add more input views to improve our reconstruction. Second, we propose a hexa-plane neural SDF representation to better recover detailed textures, geometry and material parameters. Third, we develop a novel neural directional-embedding mechanism to handle view-dependent effects. Trained on a large-scale shape and material dataset with a tailored coarse-to-fine training scheme, our model achieves compelling results. It compares favorably to optimization-based dense-view inverse rendering methods in terms of geometry and relighting accuracy, while requiring only a fraction of the inference time.

Figures (16)

• Figure 1: Given small sets of images (e.g., 4 to 8), LIRM progressively reconstructs view-dependent radiance fields, geometry and material reflectance in less than a second through a feed-forward transformer, enabling realistic rendering under novel lighting conditions. All relighting examples are reconstructed from real images in Stanford-ORB kuang2024stanford and DTC datasets Dong_2025_CVPR.
• Figure 2: The network architecture of LIRM. The inputs are masked images $\mathbf{I}^{m}$, background images to provide more lighting information $\mathbf{I}_{\text{bg}}^{m}$ and Plücker rays $(\mathbf{v}, \mathbf{v}\times \mathbf{o})^{m}$ that encodes camera intrinsics and extrinsics. These 3 images are concatenated together and turned into tokens through a simple linear layer. These tokens are sent to a self-attention transformer to update hexa-plane tokens ($\mathcal{T}_{k(+,-)}^{m}$, $k\in\{xy, xz, yz\}$) and NDE tokens ($\mathcal{E}^{m}$). We decode the 2 kinds of tokens into hexa-plane representation and NDE panoramas through linear layers, which can be used to render view dependent radiance fields and BRDF parameters through neural volume rendering. The decoded SDF volume can be used to extract accurate triangular mesh through standard marching cube.
• Figure 3: Visualization of the initial and updated reconstruction. Our simple update strategy enables our model to memorize prior reconstruction while progressively improve results.
• Figure 4: Comparisons between our tri-plane and hexa-plane reconstruction. Here we show diffuse texture reconstruction results without considering material reflection and view-dependent effects. Hexa-plane clearly recovers better texture details.
• Figure 5: Comparisons of different strategies to model view-dependent effect in a feed-forward network module.