GAINS: Gaussian-based Inverse Rendering from Sparse Multi-View Captures

TL;DR

GAINS addresses the challenge of recovering geometry, materials, and lighting from sparse multi-view captures by introducing a two-stage Gaussian-splatting pipeline. Stage I uses monocular depth/normal and diffusion priors to stabilize geometry, while Stage II combines segmentation guidance, intrinsic image decomposition, and diffusion priors to regularize material properties and lighting, enabling robust relighting and novel-view synthesis. Extensive experiments on synthetic and real datasets show substantial improvements over state-of-the-art Gaussian-based IR methods, especially under sparse-view conditions, with ablations confirming the complementary value of each prior. The work highlights the benefit of integrating multiple learning-based priors with explicit radiance representations for physically consistent inverse rendering in challenging data regimes.

Abstract

Recent advances in Gaussian Splatting-based inverse rendering extend Gaussian primitives with shading parameters and physically grounded light transport, enabling high-quality material recovery from dense multi-view captures. However, these methods degrade sharply under sparse-view settings, where limited observations lead to severe ambiguity between geometry, reflectance, and lighting. We introduce GAINS (Gaussian-based Inverse rendering from Sparse multi-view captures), a two-stage inverse rendering framework that leverages learning-based priors to stabilize geometry and material estimation. GAINS first refines geometry using monocular depth/normal and diffusion priors, then employs segmentation, intrinsic image decomposition (IID), and diffusion priors to regularize material recovery. Extensive experiments on synthetic and real-world datasets show that GAINS significantly improves material parameter accuracy, relighting quality, and novel-view synthesis compared to state-of-the-art Gaussian-based inverse rendering methods, especially under sparse-view settings. Project page: https://patrickbail.github.io/gains/

Figures (18)

• Figure 1: We introduce GAINS, GAussian-based INverse rendering from Sparse multi-view captures, which synergizes learning-based priors related to monocular depth/normal, segmentation, intrinsic image decomposition (IID), and diffusion, to better disambiguate reflectance from lighting, leading to better intrinsics, novel view synthesis and relighting compared to existing state-of-the-art approaches such as Ref-GS yao2024refGS and GI-GS chen2025gigs. Prior methods often overfit diffuse (e.g., missing yellow reflection from the ground in the first relighting example for GI-GS) and/or specular reflections (e.g., for both Ref-GS and GI-GS the reflected details remain unchanged under different lighting conditions). In contrast, GAINS improves estimation of material properties leading to better relighting in novel views.
• Figure 2: GAINS follows a two-stage inverse rendering pipeline: Stage I reconstructs geometry, and Stage II estimates material parameters and lighting. In Stage I, we enhance geometry using learning-based priors from monocular depth, normal, and diffusion predictors. In Stage II, we introduce three complementary priors: segmentation, intrinsic image decomposition (IID), and diffusion, to improve material estimation, novel-view synthesis, and relighting. Each prior provides distinct benefits: segmentation boosts cross-view consistency of specular parameters but degrades albedo; IID improves albedo accuracy but remains view-inconsistent; diffusion enhances generalization to novel view and relighting but lacks material consistency. GAINS integrates these priors to achieve stable, high-quality estimated material properties, leading to better relighting under novel views.
• Figure 3: Ablation studies on the gardensphere scene from the Ref-Real verbin2022refnerf dataset. In absence of learning-based priors (Ours w/o IID, Seg, SDS in 1st col) reflectance maps are poorly reconstructed, especially metallicity and roughness. Without the IID prior (3rd col) results in weaker specular effects (compared to the 2nd col). Without segmentation guidance (4th col) results in noise material maps across objects.
• Figure 4: Qualitative comparison of intrinsic estimation and relighting on the sedan scene from the Ref-Real dataset verbin2022refnerf reconstructed from 8 views. Column 1 shows novel-view intrinsic renderings in a 2×2 layout: (top) rendered albedo and surface normals, (bottom) specular roughness and metallicity. Columns 2–4 show relighting results under three different environment maps from novel viewpoints. GAINS recovers significantly more accurate intrinsics, enabling more realistic relighting.
• Figure 5: Qualitative comparison of albedo and roughness estimation and novel-view synthesis (NVS) on Synthetic4Relight zhang2022invrender dataset trained with 8 views. While all methods produce reasonable NVS, our method's estimates significantly better albedo and roughness than GI-GS and Ref-GS that overfit to limited training views and fail to disentangle reflectance from lighting.