Physics-Based Decomposition of Reflectance and Shading using a Single Visible-Thermal Image Pair

TL;DR

This work addresses intrinsic image decomposition (IID) from a single visible image paired with a co-registered thermal image. It develops a physics-based framework that uses heat produced by absorbed light to establish ordinal relations between visible and thermal intensities, linking them to albedo $\rho$ and shading $\eta$ through local and non-local constraints. A Double-DIP (DDIP) parameterization with edge and ordinal losses regularizes the solution without learned priors, solving for $\hat{\rho}$ and $\hat{\eta}$ by minimizing a reconstruction term plus physics-guided losses. The authors validate on synthetic and real data, introducing the VT-Intrinsic dataset of 600 visible–thermal image pairs, and demonstrate superior performance over both physics-based and learning-based baselines, highlighting the potential for scalable real-world supervision with minimal data.

Abstract

Decomposing an image into its underlying photometric factors--surface reflectance and shading--is a long-standing challenge due to the lack of extensive ground-truth data for real-world scenes. We introduce a novel physics-based approach for intrinsic image decomposition using a pair of visible and thermal images. We leverage the principle that light not reflected from an opaque surface is absorbed and detected as heat by a thermal camera. This allows us to relate the ordinalities (or relative magnitudes) between visible and thermal image intensities to the ordinalities of shading and reflectance, which enables a dense self-supervision of an optimizing neural network to recover shading and reflectance. We perform quantitative evaluations with known reflectance and shading under natural and artificial lighting, and qualitative experiments across diverse scenes. The results demonstrate superior performance over both physics-based and recent learning-based methods, providing a path toward scalable real-world data curation with supervision.

Figures (25)

• Figure 1: Through the tree's veil, sunlight weaves intricate shadows across a building façade. Visible and thermal images capture complementary cues of reflected and absorbed light. Local and non-local visible–thermal ordinalities (\ref{['sec:theory']}) reveal albedo/shading edges and point-pair ordinalities respectively, guiding an optimization using Double-DIP parameterization (\ref{['sec:method']}). Our physics-based method reconstructs the complex shading and albedo without learned priors, whereas state-of-the-art models fail (see supplementary).
• Figure 2: Printed (top) vs. projected (bottom) Roger Shepard’s illusion MindSights. Top: a printed paper lit by an incandescent bulb, where reflectance variations reveal a saxophone player. Bottom: the same pattern projected onto a uniform cardboard, where modulated shading reveals a lady’s face. This comparison highlights the albedo-shading ambiguity and motivates modeling light–heat transport: reflectance induces inverse visible–thermal ordinalities, while shading yields consistent ones. Columns 3-4 show classified albedo- / shading-dominant edges (\ref{['sec:local_constraint']}) and points of lower albedo / higher shading than (\ref{['sec:non-local_constraint']}). Our method decomposes correctly (right), whereas baselines fail (see supplementary).
• Figure 3: Examples from the visible–thermal image pairs in the VT-Intrinsic dataset, covering diverse scenes including parks, schools, cathedrals, plazas, museums, and various urban streets.
• Figure 4: Results on a color-chart scene in JoLHT-Video dataset. Our method recovers the smooth line-light shading across the color chart.
• Figure 5: Results on Painted-Mask scene in JoLHT-Video dataset. Baselines show albedo texture in shading or highlight artifacts in albedo.

Theorems & Definitions (7)

• Proposition 1: Albedo Ordinality
• Proposition 2: Shading Ordinality
• Proposition 3
• Proposition 1
• proof
• Proposition 2
• proof