Free-Form Image Inpainting with Gated Convolution

TL;DR

This work addresses free-form image inpainting by introducing gated convolution, which learns dynamic, per-location feature gating to handle irregular masks and user guidance. It pairs gated convolutions with SN-PatchGAN, a patch-based, spectral-normalized discriminator that operates on dense output maps for stable training with arbitrary masks. The authors also design a practical inpainting network architecture, a fast on-the-fly free-form mask generator, and an extension to user-guided editing via sketches, achieving superior qualitative and quantitative results on Places2 and CelebA-HQ. The approach enables robust object removal, layout modification, and interactive editing with real-time applicability, validated by user studies and ablations showing the efficacy of SN-PatchGAN and gating. The work provides open-source code and demonstrates significant practical impact for image editing tasks that require flexible, high-quality inpainting.

Abstract

We present a generative image inpainting system to complete images with free-form mask and guidance. The system is based on gated convolutions learned from millions of images without additional labelling efforts. The proposed gated convolution solves the issue of vanilla convolution that treats all input pixels as valid ones, generalizes partial convolution by providing a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers. Moreover, as free-form masks may appear anywhere in images with any shape, global and local GANs designed for a single rectangular mask are not applicable. Thus, we also present a patch-based GAN loss, named SN-PatchGAN, by applying spectral-normalized discriminator on dense image patches. SN-PatchGAN is simple in formulation, fast and stable in training. Results on automatic image inpainting and user-guided extension demonstrate that our system generates higher-quality and more flexible results than previous methods. Our system helps user quickly remove distracting objects, modify image layouts, clear watermarks and edit faces. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting

Figures (17)

• Figure 1: Illustration of partial convolution (left) and gated convolution (right).
• Figure 2: Overview of our framework with gated convolution and SN-PatchGAN for free-form image inpainting.
• Figure 3: Example cases of qualitative comparison on the Places2 and CelebA-HQ validation sets. More comparisons are included in supplementary materials due to space limit. Best viewed (e.g., shadows in uniform region) with zoom-in.
• Figure 4: Comparison of user-guided image inpainting.
• Figure 5: Object removal case study with comparison.