Dequantization and Color Transfer with Diffusion Models

TL;DR

It is shown that color palettes can make the output of the diffusion model easier to control and interpret, and overcome a common problem in existing image colorization methods that are unable to produce colors with a different luminance than the input.

Abstract

We demonstrate an image dequantizing diffusion model that enables novel edits on natural images. We propose operating on quantized images because they offer easy abstraction for patch-based edits and palette transfer. In particular, we show that color palettes can make the output of the diffusion model easier to control and interpret. We first establish that existing image restoration methods are not sufficient, such as JPEG noise reduction models. We then demonstrate that our model can generate natural images that respect the color palette the user asked for. For palette transfer, we propose a method based on weighted bipartite matching. We then show that our model generates plausible images even after extreme palette transfers, respecting user query. Our method can optionally condition on the source texture in part or all of the image. In doing so, we overcome a common problem in existing image colorization methods that are unable to produce colors with a different luminance than the input. We evaluate several possibilities for texture conditioning and their trade-offs, including luminance, image gradients, and thresholded gradients, the latter of which performed best in maintaining texture and color control simultaneously. Our method can be usefully extended to another practical edit: recoloring patches of an image while respecting the source texture. Our procedure is supported by several qualitative and quantitative evaluations.

Figures (21)

• Figure 1: We present an image dequantizer that outperforms existing image restoration models. A quantized image is shown in the first column; the second column shows the results of denoising by jiang2021towards; the third is from our procedure (which only requires the quantized image and number of palette colors as input - 4, 16, 64 here); the fourth shows GT. Our model is willing to deviate from the input to produce smoother gradients and life-like colors. Dequantization is not commonly investigated in the image restoration community - here we show why specialized dequantization models are useful, and that existing methods are not sufficient.
• Figure 2: Overview of our method. We build a controlled image synthesis pipeline to dequantize images, optionally with texture conditioning. We can apply palette transfer or inpaint patches with a user-specified color, opening new creative applications.
• Figure 3: Dequantization without texture conditioning. Our procedure outperforms existing image restoration models across a wide range of palette sizes.
• Figure 4: Dequantization with texture conditioning. Top All methods can dequantize fairly well, though $\mathsf{ours-T}$ suffers in reconstruction quality. Bottom When measuring error w.r.t. to quantized output vs. quantized input, all methods generally excel, though the fact that $\mathsf{ours-T}$ holds the least information and performs better in some cases indicates imperfections in the quantization algorithm (we use median-cut).
• Figure 5: Dequantization without texture conditioning - only the quantized image is the input (with number of colors specified in the row). Across the board, our method (fifth column) produces more natural and aesthetic results as compared with baseline denoisers chen2022simplejiang2021towards.