Thinking inside the Convolution for Image Inpainting: Reconstructing Texture via Structure under Global and Local Side

TL;DR

This paper tackles image inpainting by addressing texture feature-map loss that arises during convolutional downsampling. It introduces a reconstruction-from-structure strategy in which texture feature maps are rebuilt from structure feature maps under both global and local normalization, and couples this with a cross-layer balance module to guide upsampling. Extensive experiments on PSV, CelebA, and Places2 show consistent improvements over state-of-the-art methods in PSNR, SSIM, FID, and LPIPS on variable mask ratios and resolutions, supported by ablations that validate the necessity of global/local guidance and balanced fusion. The approach demonstrates that leveraging sparse global structure and local residual structure in a reciprocal manner yields more coherent texture preservation and semantic fidelity in high-resolution inpainting tasks.

Abstract

Image inpainting has earned substantial progress, owing to the encoder-and-decoder pipeline, which is benefited from the Convolutional Neural Networks (CNNs) with convolutional downsampling to inpaint the masked regions semantically from the known regions within the encoder, coupled with an upsampling process from the decoder for final inpainting output. Recent studies intuitively identify the high-frequency structure and low-frequency texture to be extracted by CNNs from the encoder, and subsequently for a desirable upsampling recovery. However, the existing arts inevitably overlook the information loss for both structure and texture feature maps during the convolutional downsampling process, hence suffer from a non-ideal upsampling output. In this paper, we systematically answer whether and how the structure and texture feature map can mutually help to alleviate the information loss during the convolutional downsampling. Given the structure and texture feature maps, we adopt the statistical normalization and denormalization strategy for the reconstruction guidance during the convolutional downsampling process. The extensive experimental results validate its advantages to the state-of-the-arts over the images from low-to-high resolutions including 256*256 and 512*512, especially holds by substituting all the encoders by ours. Our code is available at https://github.com/htyjers/ConvInpaint-TSGL

Figures (23)

• Figure 1: CTSDG Guo_2021_ICCV suffers from the non-ideal inpainting results due to mutual guidance between the global structure and texture feature map in decoder, where the sparse structure feature map is broken down via the fusion from the texture (b) feature map, while the texture feature map receives no effective guidance from the structure (c) feature map. The higher value within the yellow area indicates more structure or texture information from feature map. Our methods preserves more information over structure and texture feature maps. Note that the feature maps in the decoder are the result of convolutional downsampling (a), resulting into the feature map loss.
• Figure 2: Illustration of our proposed overall pipeline. Our basic idea is to reconstruct the texture feature map via the structure feature map under both global and local side during the convolutional downsampling process from encoder (ii@), which is achieved via the statistical normalization and denormalization strategy between them. We augment the guidance from global structure feature map twice to keep balance with local residual structure feature map, upon the cross-layer balance module (iii@) via the feature equalization strategy to benefit the upsampling from decoder (iv@); see Sec.\ref{['our_arch']} for more details.
• Figure 3: Comparison of the texture feature map (a) from the baseline model and the reconstructed texture feature map (b) via the structure feature map. We highlight the semantic distribution regions in both texture feature map and reconstructed texture feature map. The higher value within the yellow area indicates more structure or texture information from feature map. The KL divergence for (b) reconstructed texture feature map via structure feature map is smaller than that from (a) texture feature map, implying the intuition of reconstruction via structure feature map.
• Figure 4: Illustration of how to capture the global and local texture feature maps via global and local normalization strategies over texture feature map, i.e., (a) global normalization over the texture feature map, which captures the global statistical property for the whole feature map, named the global texture feature map. Different from that, the local normalization (b) capture the local statistical property for each position, named the local texture feature map.
• Figure 5: Illustration on how to capture the global and local residual structure feature map, respectively. (a) We employ the canny edge detector to extract the edge map as the structure information named global structure feature map. (b) To complement global structure feature map, we develop the local residual structure feature map via residual subtraction conducts over each position, i.e., the texture is unsampled with the same size as previous layer, then being subtracted to yield such residual structure feature map, which is closely related to local texture feature map.

Theorems & Definitions (1)

• Definition 1