Ultra-High-Definition Image Restoration: New Benchmarks and A Dual Interaction Prior-Driven Solution

TL;DR

This paper constructs UHD snow and rain benchmarks, named UHD-Snow and UHD-Rain, and proposes an effective UHD image restoration solution by considering gradient and normal priors in model design, thanks to these priors’ spatial and detail contributions.

Abstract

Ultra-High-Definition (UHD) image restoration has acquired remarkable attention due to its practical demand. In this paper, we construct UHD snow and rain benchmarks, named UHD-Snow and UHD-Rain, to remedy the deficiency in this field. The UHD-Snow/UHD-Rain is established by simulating the physics process of rain/snow into consideration and each benchmark contains 3200 degraded/clear image pairs of 4K resolution. Furthermore, we propose an effective UHD image restoration solution by considering gradient and normal priors in model design thanks to these priors' spatial and detail contributions. Specifically, our method contains two branches: (a) feature fusion and reconstruction branch in high-resolution space and (b) prior feature interaction branch in low-resolution space. The former learns high-resolution features and fuses prior-guided low-resolution features to reconstruct clear images, while the latter utilizes normal and gradient priors to mine useful spatial features and detail features to guide high-resolution recovery better. To better utilize these priors, we introduce single prior feature interaction and dual prior feature interaction, where the former respectively fuses normal and gradient priors with high-resolution features to enhance prior ones, while the latter calculates the similarity between enhanced prior ones and further exploits dual guided filtering to boost the feature interaction of dual priors. We conduct experiments on both new and existing public datasets and demonstrate the state-of-the-art performance of our method on UHD image low-light enhancement, dehazing, deblurring, desonwing, and deraining. The source codes and benchmarks are available at \url{https://github.com/wlydlut/UHDDIP}.

Figures (20)

• Figure 1: The UHDDIP model, when trained on low-resolution desnowing datasets CSD ChenFHTCDK21CSD, fails to remove larger snowflakes and preserve the original image's colors, while its performance remarkably improves after training on the proposed UHD-Snow dataset. Moreover, the visual results tested on the real UHD images (left three columns) and the proposed synthesized UHD-Snow images (right three columns) are consistent.
• Figure 2: The rain/snow images and their corresponding masks sampled from the proposed UHD-Snow and UHD-Rain datasets. Each dataset includes 3200 degraded/clean image pairs with 4K resolution (3000 pairs for training and 200 pairs for testing), which are synthesized snowflakes, snow streaks, and rain streaks with different densities, orientations, and locations.
• Figure 3: Statistics of our constructed UHD-Snow and UHD-Rain benchmarks.
• Figure 4: The flowchart of snow (left) and rain (right) mask generation. In the snow mask generation process, we set the parameters as follows: $50\%$ noise (amount), $15$ crystallize (cell size), $25$ pixels motion blur (distance), a threshold level range of $100$-$165$ to control the density of the snow streaks, $5$ pixels Gaussian blur (radius), and two flows of $60\%$ and $100\%$ to create snowflakes with varying transparency. For the generation of rain mask, the parameters are set: $50\%$ noise, $5$ crystallize, $200$ pixels motion blur with an angle range of $45^{\circ}$-$135^{\circ}$, a threshold level range of $55$-$67$, and $2$ pixels Gaussian blur.
• Figure 5: Overview of our Dual Interaction Prior-driven UHD restoration network (UHDDIP). UHDDIP contains two branches: (a) feature fusion and reconstruction branch in high-resolution space, which fuses low-resolution features into high-resolution space and reconstructs final images; (b) prior feature interaction branch in low-resolution space to modulate normal and gradient prior into useful features to guide high-resolution learning. We utilize NAFBlock ChenCZS22nafnet as basic feature learning units.