Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder

TL;DR

A novel pipeline that can super-resolve an input image or generate from a random noise a novel image at arbitrary scales, and is significantly better than the relevant prior-art in the inference speed and memory usage.

Abstract

Super-resolution (SR) and image generation are important tasks in computer vision and are widely adopted in real-world applications. Most existing methods, however, generate images only at fixed-scale magnification and suffer from over-smoothing and artifacts. Additionally, they do not offer enough diversity of output images nor image consistency at different scales. Most relevant work applied Implicit Neural Representation (INR) to the denoising diffusion model to obtain continuous-resolution yet diverse and high-quality SR results. Since this model operates in the image space, the larger the resolution of image is produced, the more memory and inference time is required, and it also does not maintain scale-specific consistency. We propose a novel pipeline that can super-resolve an input image or generate from a random noise a novel image at arbitrary scales. The method consists of a pretrained auto-encoder, a latent diffusion model, and an implicit neural decoder, and their learning strategies. The proposed method adopts diffusion processes in a latent space, thus efficient, yet aligned with output image space decoded by MLPs at arbitrary scales. More specifically, our arbitrary-scale decoder is designed by the symmetric decoder w/o up-scaling from the pretrained auto-encoder, and Local Implicit Image Function (LIIF) in series. The latent diffusion process is learnt by the denoising and the alignment losses jointly. Errors in output images are backpropagated via the fixed decoder, improving the quality of output images. In the extensive experiments using multiple public benchmarks on the two tasks i.e. image super-resolution and novel image generation at arbitrary scales, the proposed method outperforms relevant methods in metrics of image quality, diversity and scale consistency. It is significantly better than the relevant prior-art in the inference speed and memory usage.

Figures (19)

• Figure 1: The proposed method generates novel images and super-resolves low-resolution images at arbitrary-scales with high fidelity, diversity, and fast inference speed.
• Figure 2: Model structure comparison with IDM. The solid arrow represents the inference process, and the dotted arrow represents the error backpropagation process.
• Figure 3: Upper Part: Overall process of proposed networks. Red line is a super-resolution process, and Blue line is a generation process. Lower Left Part: Detail architecture of Implicit Neural Decoder. It contains a series of auto-decoder $\mathcal{D}_{\varphi}$ and a neural decoding function $f_{\theta}$. Lower Right Part: Pipeline of two-stage alignment process.
• Figure 4: We compare our model quantitatively with FID and SelfSSIM scores on the FFHQ datasets. The solid lines represent the FID scores of the methods that generate images of arbitrary-scale, while the dotted lines indicate the SelfSSIM scores. The '$\times$' symbol indicates the method that only generates images of a fixed scale. Our model demonstrates competitive performance in both evaluation metrics.
• Figure 5: Qualitative results on LSUN Bedroom and Church datasets. For more results, see supplementary.