ReCap: Better Gaussian Relighting with Cross-Environment Captures

TL;DR

ReCap addresses the persistent albedo-lighting ambiguity in Gaussian relighting by introducing cross-environment cross-task supervision that learns multiple lighting representations sharing a common material basis. It advances a shading function grounded in a split-sum BRDF, replaces the metallic parameter with a specular tint vector, and enforces energy conservation and saturation regularizers, all within HDR-friendly post-processing. Per-environment learnable cube maps separate diffuse and specular components, enabling direct relighting with standard HDR maps while maintaining photometric consistency across varied appearances. On an expanded RelightObj benchmark, ReCap achieves state-of-the-art relighting performance across unseen lighting and object types, while also improving normal estimation through cross-lighting constraints and demonstrating practical applicability to real captures with reasonable pose calibration.

Abstract

Accurate 3D objects relighting in diverse unseen environments is crucial for realistic virtual object placement. Due to the albedo-lighting ambiguity, existing methods often fall short in producing faithful relights. Without proper constraints, observed training views can be explained by numerous combinations of lighting and material attributes, lacking physical correspondence with the actual environment maps used for relighting. In this work, we present ReCap, treating cross-environment captures as multi-task target to provide the missing supervision that cuts through the entanglement. Specifically, ReCap jointly optimizes multiple lighting representations that share a common set of material attributes. This naturally harmonizes a coherent set of lighting representations around the mutual material attributes, exploiting commonalities and differences across varied object appearances. Such coherence enables physically sound lighting reconstruction and robust material estimation - both essential for accurate relighting. Together with a streamlined shading function and effective post-processing, ReCap outperforms all leading competitors on an expanded relighting benchmark.

Figures (16)

• Figure 1: ReCap shows a significant advantage in reconstructing environment maps with accurate tones and color fidelity. Both qualitative and quantitative assessments show that ReCap achieves more realistic and consistent relighting results under a range of unseen lighting conditions.
• Figure 2: The proposed ReCap training framework. Compared to original 3DGS kerbl20233d, each Gaussian is augmented with 3 extra material attributes. Given $k$ sets of object appearances from unknown lighting conditions as input, $k$ learnable environment maps are instantiated. Gaussian color is computed according to the shading function in the world space based on environment queries and material properties. 2D images are rasterized with standard Gaussian splatting and used for loss computation. $\mathcal{L}_{\text{image}}$: image reconstruction loss from kerbl20233d. Additional loss terms for material and geometry are not shown.
• Figure 3: With the original shading model, the shield of the helmet is falsely identified as being metallic during optimization.
• Figure 4: The same object position exhibit view-dependent and light-dependent pixel color, the later is accounted for by querying corresponding learnable environments.
• Figure 5: The comparison of estimated normal and corresponding relighting results. With single-environment captures, the highlights from the train view are falsely attributed to object property instead of lighting, passing down to relighting views. Cross-environment supervision provides more robust normal estimation and correct highlight shapes.