Do Inpainting Yourself: Generative Facial Inpainting Guided by Exemplars

TL;DR

EXE-GAN presents exemplar-guided facial inpainting by fusing input global style, stochastic latent style $\widetilde{w}$, and exemplar style $w$ to form a mixed style $\hat{w}$ across $T=18$ layers. It uses a four-component architecture (mapping network, style encoder, multi-style generator, discriminator) and optimizes with a total objective that combines $\mathcal{L}_{adv}$, $\mathcal{L}_{id}$, $\mathcal{L}_{lpips}$, and $\mathcal{L}_{attr}$ with weights, plus a spatial variant gradient backpropagation to stabilize boundaries. Evaluations on CelebA-HQ and FFHQ show superior visual quality and exemplar-consistent inpainting compared to free-form and guidance-based baselines, and demonstrate practical applications such as local facial attribute transfer, guided facial style mixing, and hairstyle editing. The approach enables interactive, exemplar-driven control over facial inpainting while preserving content, suggesting broad utility for personalized face editing and recovery tasks.

Abstract

We present EXE-GAN, a novel exemplar-guided facial inpainting framework using generative adversarial networks. Our approach can not only preserve the quality of the input facial image but also complete the image with exemplar-like facial attributes. We achieve this by simultaneously leveraging the global style of the input image, the stochastic style generated from the random latent code, and the exemplar style of exemplar image. We introduce a novel attribute similarity metric to encourage networks to learn the style of facial attributes from the exemplar in a self-supervised way. To guarantee the natural transition across the boundaries of inpainted regions, we introduce a novel spatial variant gradient backpropagation technique to adjust the loss gradients based on the spatial location. Extensive evaluations and practical applications on public CelebA-HQ and FFHQ datasets validate the superiority of EXE-GAN in terms of the visual quality in facial inpainting.

Figures (17)

• Figure 1: Facial inpainting examples using our method. Top two rows: starting with the input image (the top-left sub-image with mask), our method gradually edits the eye style (left), the mouth style (middle left), the hair style (middle), and the facial styles (right) from exemplars. Hairstyles can be edited with the insertion of basic sketches (middle). Real-world and artistic face photos can both be used to direct the inpainting of (blended) facial features in the local edited regions without affecting the visual content of the rest of the image. Bottom row: For occluded portraits with eyeglasses and masks, we perform guided facial image recovery from exemplars.
• Figure 2: Overview of our EXE-GAN framework. We employ style mixing on stochastic and exemplar style codes, and modulate them with the global style code of input image into the multi-style generator for facial inpainting. The adversarial, identity, LPIPS, and attribute losses are integrated as the overall training objective. Spatial variant gradient layers (SVGL) are utilized for natural transition across the filling boundary.
• Figure 3: Illustration of the SVGL on LPIPS and attribute losses. In forward-propagation, SVGL does not change any information for $I_{out}$. In backpropagation, gradients are re-weighted based on the spatial variant $M_{w}$ and $\overline{M}_{w}$, respectively.
• Figure 4: Quantitative comparisons between our method and the state-of-the-art free-form inpainting methods on the CelebA-HQ (top row) and FFHQ (bottom row) datasets, respectively.
• Figure 5: Qualitative comparison between our method and the state-of-the-art free-form inpainting methods.