From Posterior Sampling to Meaningful Diversity in Image Restoration

TL;DR

This paper proposes a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead, and finds the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling.

Abstract

Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here we argue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting a missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any set of samples from the posterior would be entirely dominated by (practically identical) completions of sky. However, arguably, presenting users with only one clear sky completion, along with several alternative solutions such as airships, birds, and balloons, would better outline the set of possibilities. In this paper, we initiate the study of meaningfully diverse image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful diversity. Moreover, we propose a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling. Code and examples are available at https://noa-cohen.github.io/MeaningfulDiversityInIR.

Figures (35)

• Figure 1: Approximate posterior sampling vs. meaningfully diverse sampling in image restoration. Restoration generative models aiming to sample from the posterior tend to generate images that highly resemble one another semantically (left). In contrast, the meaningful plausible solutions on the right convey a broader range of restoration possibilities. Such sets of restorations are achieved using the FPS approach explored in Sec. \ref{['sec:Baselines']}.
• Figure 2: Histograms of the projections of features from two collections of posterior samples onto their first principal component. Each collection contains $100$ reconstructions of an inpainted image. In the upper example PCA was applied on pixel space, and in the lower example on deep features of an attribute predictor. The high distribution kurtosis marked on the graphs are due to rare, yet non negligible points distant from the mean. We fit a mixture of 2 Gaussians to each distribution and plot the dominant Gaussian, to allow visual comparison of the tail.
• Figure 3: Statistics of kurtoses in posterior distributions. We calculate kurtoses values for projections of features from 54 collections of posterior samples, 100 samples each, onto their first principal component (orange). Left pane shows restored pixel values as features, while the right pane shows feature activations extracted from an attribute predictor. For comparison, we also show statistics of kurtoses of 800 multivariate Gaussians with the same dimensions (blue), each estimated from 100 samples. The non-negligible occurrence of very high kurtoses in images (compared with their Gaussian equivalents) indicates their heavy tailed distributions. Whiskers mark the $[5,95]$ percentiles.
• Figure 4: Methods for choosing a small representative set. We compare three baseline approaches for meaningfully representing a set ${\tilde{\mathcal{X}}}$ of ${\tilde{N}}=1000$ red points drawn from an imbalanced mixture of 10 Gaussians (left), by using a subset $\mathcal{X}$ of only $N=20$ points. Note how the presented approaches differ in their abilities to cover sparse and dense regions of the original set ${\tilde{\mathcal{X}}}$. In this example, ${\tilde{\mathcal{X}}}$ is dominated by the central Gaussian which contains 95% of the probability mass.
• Figure 5: Diversely sampling image restorations. Using five images to represent sets of $100$ restorations corresponding to degraded images (shown above), on images from CelebAMask-HQ (left) and PartImagenet (Right). The posterior subset (first row) is comprised of randomly drawn restoration solutions, while subsequent rows are constructed using the explored baselines.