Diverse Image Harmonization

TL;DR

This work tackles the inherent uncertainty in image harmonization arising from unknown foreground reflectance by introducing a reflectance-guided harmonization framework and a Diverse Reflectance Generation Network. When the original foreground image is available, ground-truth foreground reflectance $A_{gt}$ extracted via Retinex-based decomposition informs the harmonization, yielding results closer to ground-truth. When $A_{gt}$ is unavailable, a two-branch network learns a distribution of plausible foreground reflectances $A_{pre}$, enabling multiple harmonious outcomes; an unsupervised branch with adversarial training promotes realistic reflectances. Evaluations on iHarmony4 and RealHM demonstrate improved fidelity with $A_{gt}$ guidance and show that diverse $A_{pre}$ lead to multiple plausible harmonizations, validated by ablations of network components and losses.

Abstract

Image harmonization aims to adjust the foreground illumination in a composite image to make it harmonious. The existing harmonization methods can only produce one deterministic result for a composite image, ignoring that a composite image could have multiple plausible harmonization results due to multiple plausible reflectances. In this work, we first propose a reflectance-guided harmonization network, which can achieve better performance with the guidance of ground-truth foreground reflectance. Then, we also design a diverse reflectance generation network to predict multiple plausible foreground reflectances, leading to multiple plausible harmonization results. The extensive experiments on the benchmark datasets demonstrate the effectiveness of our method.

Figures (6)

• Figure 1: An example of multiple plausible harmonization results when the foreground reflectance is uncertain.
• Figure 2: In the left part, we show two example triplets of original foreground image, background, and composite image on iHarmony4 dovenet and RealHM ssh. In the right part, we use pretrained RelightingNet albedo to extract the ground-truth foreground reflectance map $\bm{A}_{gt}$ from the original foreground image $\bm{I}_o$.
• Figure 3: The left part shows the diverse reflectance generation network with two branches: supervised and unsupervised branch. In the supervised branch, we extract the guidance code from $\bm{I}_o$, which guides the network $G$ to reconstruct the ground-truth foreground reflectance map. In the unsupervised branch, we extract the guidance code from the foreground region, which guides $G$ to predict plausible foreground reflectance maps. The right part shows the reflectance-guided harmonization network, in which foreground reflectance map is appended to the input to guide the harmonization process.
• Figure 4: The harmonization results of iSSAM iSSAM and PCT-Net PCT-Net on iHarmony4 dataset when using or without using ground-truth foreground reflectance map.
• Figure 5: The harmonization results of iSSAM iSSAM and PCT-Net PCT-Net on realHM dataset when using or without using ground-truth foreground reflectance map.