Don't Look into the Dark: Latent Codes for Pluralistic Image Inpainting

TL;DR

The proposed method outperforms strong baselines in both visual quality and diversity metrics as the proposed method learns latent priors, discretized as tokens, by only performing computations at the visible locations of the image.

Abstract

We present a method for large-mask pluralistic image inpainting based on the generative framework of discrete latent codes. Our method learns latent priors, discretized as tokens, by only performing computations at the visible locations of the image. This is realized by a restrictive partial encoder that predicts the token label for each visible block, a bidirectional transformer that infers the missing labels by only looking at these tokens, and a dedicated synthesis network that couples the tokens with the partial image priors to generate coherent and pluralistic complete image even under extreme mask settings. Experiments on public benchmarks validate our design choices as the proposed method outperforms strong baselines in both visual quality and diversity metrics.

Figures (14)

• Figure 1: Inpainting results on the Places Dataset zhou2017places (first two rows) and the CelebA-HQ Dataset karras2018progressive (third row). Our method is able to diversely complete partial image with free-form, large holes with state-of-the-art visual quality.
• Figure 2: Overall pipeline of our method. $E_{rst}$ denotes our proposed restrictive encoder that predicts partial tokens from the source image (see Section \ref{['sec:encoder']}). The grey square space in the figure denotes missing tokens, which are iteratively predicted by a bidirectional transformer (see Section \ref{['sec:transformer']}). $E_{prt}$ denotes an encoder with partial convolution layers, which processes the source image into complementary features to the predicted tokens. The coupled features are decoded into a complete image by a generate $G$ (see Section \ref{['sec:decoder']}).
• Figure 3: A visualization of mask down-sampling, shown on a 16x16 grid on the third column, from different $\alpha$ values following Equation \ref{['eq:restrictive2']}. Smaller $\alpha$ values (top two rows) lead the restrictive encoder to predict tokens for more small mask areas (marked by the red pixels). Larger $\alpha$ is undesirable (bottom two rows) as it unnecessarily discards useful information from the image, leading to more inconsistent inpainting results.
• Figure 4: A visual comparison between the decoder designs. A. Directly decoding the predicted latent codes $Z$ with the restrictive encoder $E$ and transformer $T$, and B. its composition with the source image $X_M$. C. Our proposed decoding design, where partial image priors $E_{prt}(X_M)$ are composed with $Z$ through a composition function $f$ described in Equation.\ref{['eq:decoder_1']}-\ref{['eq:decoder_2']}.
• Figure 5: Visual examples on inpainting with both the random masks (upper half) and the challenging large box mask (lower half), compared to the selected baseline methods.