Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Painting Outside the Box: Image Outpainting with GANs

Mark Sabini, Gili Rusak

TL;DR

This work tackles image outpainting by framing it as a GAN-based extrapolation task using a DCGAN-style generator and both global and local discriminators. A three-phase training schedule plus dilated convolutions yields stable learning and a sufficiently large receptive field to plausibly extend 128×128 images, with additional postprocessing to blend the extrapolated region. The approach demonstrates both qualitative realism and quantitative feasibility (RMSE over the masked region) on Places365, including a sanity check with overfitting on a single image and a recursive outpainting extension. The findings suggest practical pathways for panorama and texture generation, while outlining future directions such as perceptual losses, partial convolutions, and video extension for broader applicability.

Abstract

The challenging task of image outpainting (extrapolation) has received comparatively little attention in relation to its cousin, image inpainting (completion). Accordingly, we present a deep learning approach based on Iizuka et al. for adversarially training a network to hallucinate past image boundaries. We use a three-phase training schedule to stably train a DCGAN architecture on a subset of the Places365 dataset. In line with Iizuka et al., we also use local discriminators to enhance the quality of our output. Once trained, our model is able to outpaint $128 \times 128$ color images relatively realistically, thus allowing for recursive outpainting. Our results show that deep learning approaches to image outpainting are both feasible and promising.

Painting Outside the Box: Image Outpainting with GANs

TL;DR

This work tackles image outpainting by framing it as a GAN-based extrapolation task using a DCGAN-style generator and both global and local discriminators. A three-phase training schedule plus dilated convolutions yields stable learning and a sufficiently large receptive field to plausibly extend 128×128 images, with additional postprocessing to blend the extrapolated region. The approach demonstrates both qualitative realism and quantitative feasibility (RMSE over the masked region) on Places365, including a sanity check with overfitting on a single image and a recursive outpainting extension. The findings suggest practical pathways for panorama and texture generation, while outlining future directions such as perceptual losses, partial convolutions, and video extension for broader applicability.

Abstract

The challenging task of image outpainting (extrapolation) has received comparatively little attention in relation to its cousin, image inpainting (completion). Accordingly, we present a deep learning approach based on Iizuka et al. for adversarially training a network to hallucinate past image boundaries. We use a three-phase training schedule to stably train a DCGAN architecture on a subset of the Places365 dataset. In line with Iizuka et al., we also use local discriminators to enhance the quality of our output. Once trained, our model is able to outpaint color images relatively realistically, thus allowing for recursive outpainting. Our results show that deep learning approaches to image outpainting are both feasible and promising.

Paper Structure

This paper contains 18 sections, 4 equations, 9 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Preprocessing
  5. Methods
  6. Training Pipeline
  7. Training Schedule
  8. Network Architecture
  9. Evaluation Metrics
  10. Results
  11. Overfitting on a Single Image
  12. Outpainting on Places365
  13. Local Discriminator
  14. Significance of Dilated Convolutions
  15. Conclusions
  16. ...and 3 more sections

Figures (9)

  • Figure 1: City image and sample images from Places365 dataset.
  • Figure 2: Training pipeline
  • Figure 3: Generator, $G$
  • Figure 4: Discriminator, $D$
  • Figure 5: Outpainting results for a sample of held-out images in the validation set, shown alongside the original ground truth. These results were produced without the aid of local discriminators. The RMSE between each $(\textrm{ground truth}, \textrm{output})$ pair is displayed below each column.
  • ...and 4 more figures