Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global-Local Stepwise Generative Network for Ultra High-Resolution Image Restoration

Xin Feng, Haobo Ji, Wenjie Pei, Fanglin Chen, Guangming Lu

TL;DR

Extensive experiments across three typical tasks for image background restoration, including image reflection removal, image rain streak removal and image dehazing, show that the GLSGN consistently outperforms state-of-the-art methods.

Abstract

While the research on image background restoration from regular size of degraded images has achieved remarkable progress, restoring ultra high-resolution (e.g., 4K) images remains an extremely challenging task due to the explosion of computational complexity and memory usage, as well as the deficiency of annotated data. In this paper we present a novel model for ultra high-resolution image restoration, referred to as the Global-Local Stepwise Generative Network (GLSGN), which employs a stepwise restoring strategy involving four restoring pathways: three local pathways and one global pathway. The local pathways focus on conducting image restoration in a fine-grained manner over local but high-resolution image patches, while the global pathway performs image restoration coarsely on the scale-down but intact image to provide cues for the local pathways in a global view including semantics and noise patterns. To smooth the mutual collaboration between these four pathways, our GLSGN is designed to ensure the inter-pathway consistency in four aspects in terms of low-level content, perceptual attention, restoring intensity and high-level semantics, respectively. As another major contribution of this work, we also introduce the first ultra high-resolution dataset to date for both reflection removal and rain streak removal, comprising 4,670 real-world and synthetic images. Extensive experiments across three typical tasks for image background restoration, including image reflection removal, image rain streak removal and image dehazing, show that our GLSGN consistently outperforms state-of-the-art methods.

Global-Local Stepwise Generative Network for Ultra High-Resolution Image Restoration

TL;DR

Extensive experiments across three typical tasks for image background restoration, including image reflection removal, image rain streak removal and image dehazing, show that the GLSGN consistently outperforms state-of-the-art methods.

Abstract

While the research on image background restoration from regular size of degraded images has achieved remarkable progress, restoring ultra high-resolution (e.g., 4K) images remains an extremely challenging task due to the explosion of computational complexity and memory usage, as well as the deficiency of annotated data. In this paper we present a novel model for ultra high-resolution image restoration, referred to as the Global-Local Stepwise Generative Network (GLSGN), which employs a stepwise restoring strategy involving four restoring pathways: three local pathways and one global pathway. The local pathways focus on conducting image restoration in a fine-grained manner over local but high-resolution image patches, while the global pathway performs image restoration coarsely on the scale-down but intact image to provide cues for the local pathways in a global view including semantics and noise patterns. To smooth the mutual collaboration between these four pathways, our GLSGN is designed to ensure the inter-pathway consistency in four aspects in terms of low-level content, perceptual attention, restoring intensity and high-level semantics, respectively. As another major contribution of this work, we also introduce the first ultra high-resolution dataset to date for both reflection removal and rain streak removal, comprising 4,670 real-world and synthetic images. Extensive experiments across three typical tasks for image background restoration, including image reflection removal, image rain streak removal and image dehazing, show that our GLSGN consistently outperforms state-of-the-art methods.
Paper Structure (18 sections, 11 equations, 13 figures, 9 tables)

This paper contains 18 sections, 11 equations, 13 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Global-Local Stepwise Generative Network
  4. Global-Local Stepwise Generative Framework
  5. Global-Local Consistency between Pathways for Collaborative Image Restoration
  6. Jointly Supervised Parameter Learning
  7. UHR4K Dataset
  8. Image Reflection Removal
  9. Image Rain Streak Removal
  10. Comparison with Existing Datasets
  11. Experiments
  12. Implementation Details
  13. Ablation Study
  14. Comparison with State-of-the-art Methods
  15. Experiments on image reflection removal
  16. ...and 3 more sections

Figures (13)

  • Figure 1: Proposed global-local stepwise generative network for ultra high-resolution image restoration. 'Cat' denotes concatenation of patches to intact feature maps, and $\otimes$ is the element-wise multiplication.
  • Figure 2: (a) The structure of perceptual attention consistency (PAC) mechanism. (b) The structure of spatial attention.
  • Figure 3: An example of reflection removal for our proposed patch normalization(PN). (a) The visualization of equation \ref{['equ1:PN']}. (b) The restored result without patch normalization. (c) The restored result by patch normalization.
  • Figure 4: Laplacian Pyramid for reconstructing full resolution ultra high-resolution background image. The green arrow denotes the mask updating steps.
  • Figure 5: The 4K resolution samples from our UHR4K dataset. Top: images with reflection or rain streak. Bottom: the corresponding groundtruth background images.
  • ...and 8 more figures