Retrieval Augmented Image Harmonization

TL;DR

This work tackles the ill-posedness of image harmonization when the background lacks content similar to the foreground and when attention is corrupted by irrelevant regions. It introduces Raiha, a retrieval-augmented framework that jointly retrieves illumination- and content-consistent references and uses a semantic-guided fusion module to constrain attention to semantically related regions. Key contributions include a harmonization-oriented retrieval pipeline (foreground content retrieval and background illumination retrieval), a semantic-guided fusion module, and a data augmentation strategy that leverages non-reference data to train robustly. Experiments on iHarmony4 and the proposed Raiharmony4 dataset demonstrate state-of-the-art performance under both non-reference and retrieval-augmented settings, highlighting the practical impact of combining retrieval with harmonization for visually coherent composites.

Abstract

When embedding objects (foreground) into images (background), considering the influence of photography conditions like illumination, it is usually necessary to perform image harmonization to make the foreground object coordinate with the background image in terms of brightness, color, and etc. Although existing image harmonization methods have made continuous efforts toward visually pleasing results, they are still plagued by two main issues. Firstly, the image harmonization becomes highly ill-posed when there are no contents similar to the foreground object in the background, making the harmonization results unreliable. Secondly, even when similar contents are available, the harmonization process is often interfered with by irrelevant areas, mainly attributed to an insufficient understanding of image contents and inaccurate attention. As a remedy, we present a retrieval-augmented image harmonization (Raiha) framework, which seeks proper reference images to reduce the ill-posedness and restricts the attention to better utilize the useful information. Specifically, an efficient retrieval method is designed to find reference images that contain similar objects as the foreground while the illumination is consistent with the background. For training the Raiha framework to effectively utilize the reference information, a data augmentation strategy is delicately designed by leveraging existing non-reference image harmonization datasets. Besides, the image content priors are introduced to ensure reasonable attention. With the presented Raiha framework, the image harmonization performance is greatly boosted under both non-reference and retrieval-augmented settings. The source code and pre-trained models will be publicly available.

Figures (11)

• Figure 1: When the background contains similar content to the foreground, it can provide internal guidance for adjusting foreground appearance. Conversely, image harmonization becomes ill-posed. Providing references that have similar content to the foreground can alleviate this issue. However, while the illumination between the reference and background is different, it fails to provide reliable guidance. Thus, when a reference has a consistent illumination condition with the background, the appearance of the foreground-similar content can provide reliable guidance, as it has a similar appearance to the target image.
• Figure 2: (a) presents the framework of our proposed Raiha, which solves the ill-posed issues when similar content is unavailable in the background $\bm{\mathit{x}}_b$. Firstly, given a composite image $\tilde{\bm{\mathit{x}}}$, we design a harmonization-oriented image retrieval method to produce proper a reference $\bm{\mathit{x}}_r$ from the gallery $\mathcal{G}$. Specifically, a foreground content retrieval is designed to construct the gallery $\mathcal{G}_{sc}$, in which images contain similar content with foreground $\bm{\mathit{x}}_f$. The background illumination retrieval is employed to construct the gallery $\mathbf{G}_{sc\&ci}$, in which images contain consistent illumination with the background $\bm{\mathit{x}}_b$. The Semantic-Guided Fusion module (c) is also designed to enable Raiha to effectively utilize the reference. Finally, Raiha takes both $\tilde{\bm{\mathit{x}}}$ and the reference $\bm{\mathit{x}}_r$ as inputs to generate visually pleasant results $\hat{\bm{\mathit{x}}}$.
• Figure : Input
• Figure : Input
• Figure : DCCF