Stronger Semantic Encoders Can Harm Relighting Performance: Probing Visual Priors via Augmented Latent Intrinsics

TL;DR

This work shows that stronger semantic encoders can degrade relighting performance, revealing a trade-off between semantic abstraction and photometric fidelity. It introduces Augmented Latent Intrinsics (ALI), a three-stage method that fuses dense pixel-aligned visual priors with latent intrinsic representations and uses self-supervision to learn from unlabeled real-world pairs. Through Stage I augmentation, Stage II decoder alignment, and Stage III self-refinement, ALI achieves state-of-the-art diffusion-based relighting on the MIIW benchmark, especially for challenging glossy and specular materials. The results suggest that carefully designed prior integration, rather than mere scaling of semantic encoders, is key for physically grounded generative tasks in graphics and computational photography.

Abstract

Image-to-image relighting requires representations that disentangle scene properties from illumination. Recent methods rely on latent intrinsic representations but remain under-constrained and often fail on challenging materials such as metal and glass. A natural hypothesis is that stronger pretrained visual priors should resolve these failures. We find the opposite: features from top-performing semantic encoders often degrade relighting quality, revealing a fundamental trade-off between semantic abstraction and photometric fidelity. We study this trade-off and introduce Augmented Latent Intrinsics (ALI), which balances semantic context and dense photometric structure by fusing features from a pixel-aligned visual encoder into a latent-intrinsic framework, together with a self-supervised refinement strategy to mitigate the scarcity of paired real-world data. Trained only on unlabeled real-world image pairs and paired with a dense, pixel-aligned visual prior, ALI achieves strong improvements in relighting, with the largest gains on complex, specular materials. Project page: https:\\augmented-latent-intrinsics.github.io

Figures (8)

• Figure 1: Our Three-Stage Training Pipeline for Augmented Latent Intrinsics (ALI). Our method progressively adapts a pretrained visual encoder and fine-tunes a generative decoder for high-fidelity, unsupervised relighting. Stage I: Augmenting Latent Intrinsics. We inject semantic features from a frozen vision encoder into the intrinsics encoder. This creates our semantically-enriched ALI, which better disentangles scene properties from illumination. Stage II: Aligning the Generative Decoder. With the encoder fixed, we fine-tune the LumiNet diffusion decoder to condition on the new ALI representation, aligning the generator with the scene's improved physical understanding. Stage III: Self-Refinement. We generate a "Lighting Zoo" (examples in supplementary) of pseudo-relit images to overcome data scarcity. These synthetic images serve as new inputs, with the original real image as the ground truth. This self-supervision trains the network to ignore artifacts and focus on essential structural properties, improving realism for in-the-wild images.
• Figure 2: Qualitative comparison of relighting methods on challenging MIIW test scenes. Our image-to-image approach produces more physically plausible results than competing methods, many of which rely on privileged information like GT light maps, G-buffers, or albedo. The task is to relight the Input scene using the illumination from the Target lighting image. In the top row, competing methods render the metallic toaster with faint or blurry specular highlights, while our method produces sharp, plausible reflections that are much closer to the ground truth (GT). Similarly, in the second row, all baseline methods generate incorrect or missing shadows of the orange cereal box, and LumiNet also blurs the text on the packaging. Our method preserves these details and renders more realistic shadows. Finally, in the third row, all baselines struggle to render the complex transparency and caustic light effects of the bottles, a common failure case that our method addresses with a more plausible result. The results for UniRelight were provided by the authors. Our approach demonstrates that augmenting dense visual priors with latent intrinsics improves a model's relightability and its understanding of light transport, all without 3D or inverse graphics supervision. Red and green markers highlight specific failures in competing methods and successes in our method, respectively. Best viewed on screen with zoom.
• Figure 3: Relighting comparison across two real-world images, each shown under two target illuminations (Light 1: lamp-dominated; Light 2: sunlight through windows). IC-Light IC-Light produces stylized results with exaggerated glow and artifacts that diverge from the targets. Latent-intrinsics Latent-Intrinsic captures some variation but yields low-contrast, flattened illumination with weak directionality. LumiNet LumiNet better matches global tone but remains overly diffuse, often missing dominant light sources and underestimating cast shadows and highlight localization. Ours preserves material detail and transfers both global and directional lighting, producing images that most closely match the targets lighting.
• Figure 4: Multi-stage ablation.Top: Compared to LumiNet, our Stage I improves fine geometry details. Adding Stage II sharpens directional cues and specular effects, while the full pipeline (Stage I&II&III) produces the closest match to ground truth, with accurate shadows, highlights, and material fidelity. This progression illustrates how each stage contributes complementary improvements, consistent with the quantitative gains in Tab. \ref{['tab:metarial_metrics']}. Bottom: LumiNet produces flat illumination with weak lamp cues. Stage I introduces coarse global tone but has color shift, Stage II suppresses these effects, and the full pipeline (Stage I&II&III) yields the most faithful transfer: interior warmth from the lamp is preserved while maintaining the outdoor scene, closely matching the target lighting. This demonstrates that our stage-wise design generalizes to unconstrained real-world images.
• Figure 5: Lighting interpolation and diversity.Top: Generated images showing a smooth interpolation between two lighting codes. Note the plausible evolution of directional lighting, including the progressive appearance of sharp specular highlights on the toaster and caustic effects from the bottle. Bottom: In-the-wild relighting results using lighting codes sampled from random, unpaired images. Our method produces a diverse range of distinct illumination effects, plausible altering the glossy reflections on the dining table and the ambient lighting in the living room.