Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Region-wise Generative Adversarial ImageInpainting for Large Missing Areas

Yuqing Ma, Xianglong Liu, Shihao Bai, Lei Wang, Aishan Liu, Dacheng Tao, Edwin Hancock

TL;DR

This work addresses the challenge of inpainting large, potentially contiguous missing regions by introducing a region-wise generative adversarial framework. It uses region-specific convolutions to treat existing and missing regions differently, a non-local correlation (correlation) loss to capture cross-patch relationships, and a region-wise discriminator to curb artifacts, achieving superior results on large holes across multiple datasets. The approach yields semantically coherent and visually realistic restorations, outperforming state-of-the-art methods in both contiguity scenarios and across diverse image domains. It also demonstrates practical utility in tasks like unwanted object removal, indicating broad applicability in image editing and restoration.

Abstract

Recently deep neutral networks have achieved promising performance for filling large missing regions in image inpainting tasks. They usually adopted the standard convolutional architecture over the corrupted image, leading to meaningless contents, such as color discrepancy, blur and artifacts. Moreover, most inpainting approaches cannot well handle the large continuous missing area cases. To address these problems, we propose a generic inpainting framework capable of handling with incomplete images on both continuous and discontinuous large missing areas, in an adversarial manner. From which, region-wise convolution is deployed in both generator and discriminator to separately handle with the different regions, namely existing regions and missing ones. Moreover, a correlation loss is introduced to capture the non-local correlations between different patches, and thus guides the generator to obtain more information during inference. With the help of our proposed framework, we can restore semantically reasonable and visually realistic images. Extensive experiments on three widely-used datasets for image inpainting tasks have been conducted, and both qualitative and quantitative experimental results demonstrate that the proposed model significantly outperforms the state-of-the-art approaches, both on the large continuous and discontinuous missing areas.

Region-wise Generative Adversarial ImageInpainting for Large Missing Areas

TL;DR

This work addresses the challenge of inpainting large, potentially contiguous missing regions by introducing a region-wise generative adversarial framework. It uses region-specific convolutions to treat existing and missing regions differently, a non-local correlation (correlation) loss to capture cross-patch relationships, and a region-wise discriminator to curb artifacts, achieving superior results on large holes across multiple datasets. The approach yields semantically coherent and visually realistic restorations, outperforming state-of-the-art methods in both contiguity scenarios and across diverse image domains. It also demonstrates practical utility in tasks like unwanted object removal, indicating broad applicability in image editing and restoration.

Abstract

Recently deep neutral networks have achieved promising performance for filling large missing regions in image inpainting tasks. They usually adopted the standard convolutional architecture over the corrupted image, leading to meaningless contents, such as color discrepancy, blur and artifacts. Moreover, most inpainting approaches cannot well handle the large continuous missing area cases. To address these problems, we propose a generic inpainting framework capable of handling with incomplete images on both continuous and discontinuous large missing areas, in an adversarial manner. From which, region-wise convolution is deployed in both generator and discriminator to separately handle with the different regions, namely existing regions and missing ones. Moreover, a correlation loss is introduced to capture the non-local correlations between different patches, and thus guides the generator to obtain more information during inference. With the help of our proposed framework, we can restore semantically reasonable and visually realistic images. Extensive experiments on three widely-used datasets for image inpainting tasks have been conducted, and both qualitative and quantitative experimental results demonstrate that the proposed model significantly outperforms the state-of-the-art approaches, both on the large continuous and discontinuous missing areas.

Paper Structure

This paper contains 23 sections, 7 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Methods
  4. Deep Generative Methods
  5. Synthesising Realistic Contents
  6. Inferring High Frequency Details
  7. Filling Irregular Holes
  8. The Approach
  9. The Adversarial Inpainting Framework
  10. Generating Region-wise Contents
  11. Inferring Missing Contents via Correlations
  12. Eliminating large area artifacts
  13. The Formulation and Optimization
  14. Experiments
  15. Datasets and Protocols
  16. ...and 8 more sections

Figures (8)

  • Figure 1: Image inpainting results for large missing areas (discontiguous at the top row, and contiguous at the bottom row), using EdgeConnect (EC) nazeri2019edgeconnect , our previous model Region-wise Encoder-Decoder (RED) ma2019inpainting and our proposed method on street view image.
  • Figure 2: The architecture of our proposed region-wise adversarial image inpainting framework.
  • Figure 3: Effect of the perceptual loss $\mathcal{L}_p$ in EC and PConv.
  • Figure 4: Results of inpainting on the large contiguous and discontiguous missing areas generated by masking randomly. (a) the input incomplete images, (b) results using standard convolutions instead of our region-wise convolutions, (c) results of model trained without our correlation loss $\mathcal{L}_c$, (d) results of model trained with $\mathcal{L}_c, \mathcal{L}_s$ at the networks, (e) results of the semantic inferring network, (f) results of model trained without adversarial loss, namely REDma2019inpainting and (g) results of our full model.
  • Figure 5: Qualitative comparisons between different methods on CelebA-HQ
  • ...and 3 more figures