IDT: A Physically Grounded Transformer for Feed-Forward Multi-View Intrinsic Decomposition

TL;DR

The paper tackles multi-view intrinsic image decomposition by introducing Intrinsic Decomposition Transformer (IDT), a feed-forward framework that jointly reasons over multiple views using a transformer while enforcing a physically grounded image formation model. By explicitly modeling diffuse reflectance, diffuse shading, and specular shading, IDT achieves view-consistent intrinsic factors in a single pass, aided by appearance adapters and scene-conditioned cross-attention. Quantitative and qualitative results on Hypersim and InteriorVerse show superior multi-view consistency and cleaner decomposition compared to single-view and diffusion-based baselines, with robust reconstruction. This approach enables scalable, interpretable material and illumination reasoning across views, with broad implications for relighting and multi-view scene understanding.

Abstract

Intrinsic image decomposition is fundamental for visual understanding, as RGB images entangle material properties, illumination, and view-dependent effects. Recent diffusion-based methods have achieved strong results for single-view intrinsic decomposition; however, extending these approaches to multi-view settings remains challenging, often leading to severe view inconsistency. We propose \textbf{Intrinsic Decomposition Transformer (IDT)}, a feed-forward framework for multi-view intrinsic image decomposition. By leveraging transformer-based attention to jointly reason over multiple input images, IDT produces view-consistent intrinsic factors in a single forward pass, without iterative generative sampling. IDT adopts a physically grounded image formation model that explicitly decomposes images into diffuse reflectance, diffuse shading, and specular shading. This structured factorization separates Lambertian and non-Lambertian light transport, enabling interpretable and controllable decomposition of material and illumination effects across views. Experiments on both synthetic and real-world datasets demonstrate that IDT achieves cleaner diffuse reflectance, more coherent diffuse shading, and better-isolated specular components, while substantially improving multi-view consistency compared to prior intrinsic decomposition methods.

Figures (4)

• Figure 1: Teaser illustration of IDT inference. IDT jointly reasons over multiple views to decompose each image into diffuse reflectance, diffuse shading, and specular shading in a single feed-forward pass. These intrinsic factors faithfully reconstruct the original appearance and support relighting by altering illumination while maintaining consistent material properties across views.
• Figure 2: Overview of the IDT pipeline. Given multiple images of a static scene, IDT first aggregates cross-view information using a multi-view transformer encoder. The shared latent tokens are then selectively routed by factor-specific appearance adapters to predict view-invariant albedo, view-dependent diffuse and specular shading, and a shared scene-level illumination representation. All intrinsic factors are inferred in a single feed-forward pass and are jointly constrained by a physically grounded image formation model.
• Figure 3: Illustration of the physically grounded image formation model. For each view, the observed image is decomposed into a view-invariant diffuse reflectance, a view-dependent shading term modeling Lambertian illumination, and an additive view-dependent specular component capturing non-Lambertian effects. This formulation yields a multiplicative separation between albedo and diffuse shading and explicitly isolates specular appearance to prevent view-dependent effects from leaking into material properties.
• Figure 4: Qualitative multi-view intrinsic decomposition results on synthetic and real-world scenes. Given three input images from different viewpoints, IDT performs feed-forward joint inference to produce view-consistent diffuse reflectance, diffuse shading, specular shading, and surface normals. The results highlight effective separation of Lambertian and non-Lambertian effects across views.