Boundary-aware Decoupled Flow Networks for Realistic Extreme Rescaling

TL;DR

The proposed Boundary-aware Decoupled Flow Networks (BDFlow) first decouples the high-frequency information into semantic high-frequency that adheres to a Boundary distribution and non-semantic high-frequency counterpart that adheres to a Gaussian distribution.

Abstract

Recently developed generative methods, including invertible rescaling network (IRN) based and generative adversarial network (GAN) based methods, have demonstrated exceptional performance in image rescaling. However, IRN-based methods tend to produce over-smoothed results, while GAN-based methods easily generate fake details, which thus hinders their real applications. To address this issue, we propose Boundary-aware Decoupled Flow Networks (BDFlow) to generate realistic and visually pleasing results. Unlike previous methods that model high-frequency information as standard Gaussian distribution directly, our BDFlow first decouples the high-frequency information into \textit{semantic high-frequency} that adheres to a Boundary distribution and \textit{non-semantic high-frequency} counterpart that adheres to a Gaussian distribution. Specifically, to capture semantic high-frequency parts accurately, we use Boundary-aware Mask (BAM) to constrain the model to produce rich textures, while non-semantic high-frequency part is randomly sampled from a Gaussian distribution.Comprehensive experiments demonstrate that our BDFlow significantly outperforms other state-of-the-art methods while maintaining lower complexity. Notably, our BDFlow improves the PSNR by 4.4 dB and the SSIM by 0.1 on average over GRAIN, utilizing only 74% of the parameters and 20% of the computation. The code will be available at https://github.com/THU-Kingmin/BAFlow.

Figures (6)

• Figure 1: Visual quality of various IRN-based and GAN-based methods, including (b) GLEAN, (c) IRN, and (d) GRAIN. Existing methods produce (c) over-smoothed results or (b) and (d) fake details. By contrast, our method can generate visually pleasing results with sharper details.
• Figure 2: The comparison of different modeling approaches to $Z_{\text{GT}}$. We statistics the distribution of high-frequency information for the DIV2K and CelebA, $Z_{\text{GT}}$: $\mu = 0$, $\sigma^2 = 0.2$. IRN is non-decoupled and models $Z_{\text{GT}}$ as a standard Gaussian distribution, $Z_{\text{IRN}}$. HCFlow is also non-decoupled and learns biased estimation, $Z_{\text{HCFlow}}$. Our BDFlow is decoupled and models the high-frequency information as semantic distribution, $B$ and non-semantic distribution, $Z_{\text{BDFlow}}$.
• Figure 3: The overall architecture of our Boundary-aware Decoupled Flow Networks (BDFlow), which comprises Haar Wavelet Blocks and Flow Networks, which further consist of multiple stacked Invertible Blocks (InvBlock). Each InvBlock incorporates three convolutional transformation functions $\phi (\cdot)$, $\rho (\cdot)$, and $\eta (\cdot)$, which enhance the nonlinear representation. $Z_n$ denotes non-semantic high-frequency information that adheres to a Gaussian distribution, while $B_n$ corresponds to semantic high-frequency information that adheres to a Boundary distribution.
• Figure 4: Visual results of rescaling the HR images with $1024 \times 1024$. The LR images are $16 \times 16$ for rescaling factor $64$. BDFlow recovers rich textures and realistic details, leading to better recovery performance.
• Figure 5: Visual results of rescaling the HR images with $1024 \times 1024$. The LR images are $16 \times 16$ for $\times 64$. BDFlow helps to restore more realistic and clear hair details.