Prototype Clustered Diffusion Models for Versatile Inverse Problems

TL;DR

This work addresses inverse problems under capricious, real-world deterioration by introducing restorer guidance, which replaces the conventional measurement-based data-consistency likelihood with a restoration-based likelihood derived from off-the-shelf restoration models. The key idea is to use the opposite probabilistic direction to form a flexible, clustered deterioration framework, enabling posterior sampling that can accommodate diverse disturbances without retraining the diffusion model. The authors develop three extensions—gradient orientation, restorer traveling, and measurement boosting—and demonstrate improved performance on image dehazing, rain streak removal, and motion deblurring, including out-of-distribution scenarios. The approach unifies Bayesian diffusion with restoration-based priors, offers deterioration-control mechanisms, and provides a practical, zero-shot solver that leverages existing restoration tools to enhance sample quality and robustness in challenging inverse problems.

Abstract

Diffusion models have made remarkable progress in solving various inverse problems, attributing to the generative modeling capability of the data manifold. Posterior sampling from the conditional score function enable the precious data consistency certified by the measurement-based likelihood term. However, most prevailing approaches confined to the deterministic deterioration process of the measurement model, regardless of capricious unpredictable disturbance in real-world sceneries. To address this obstacle, we show that the measurement-based likelihood can be renovated with restoration-based likelihood via the opposite probabilistic graphic direction, licencing the patronage of various off-the-shelf restoration models and extending the strictly deterministic deterioration process to adaptable clustered processes with the supposed prototype, in what we call restorer guidance. Particularly, assembled with versatile prototypes optionally, we can resolve inverse problems with bunch of choices for assorted sample quality and realize the proficient deterioration control with assured realistic. We show that our work can be formally analogous to the transition from classifier guidance to classifier-free guidance in the field of inverse problem solver. Experiments on multifarious inverse problems demonstrate the effectiveness of our method, including image dehazing, rain streak removal, and motion deblurring.

Figures (13)

• Figure 1: Visual illustration of the data consistency (likelihood) term in prevailing diffusion-based inverse problem solvers. Compared to the deterministic deterioration process of the measurement-based likelihood, the restoration-based likelihood is more adaptable for versatile inverse problems with capricious unpredictable disturbance, such as varying weather conditions and irregular manual disruption, by virtue of the augmented prototype clustered deterioration processes.
• Figure 2: Probabilistic graphic model of the likelihood term. The direction of restoration-based likelihood is opposite to the prevailing measurement-based likelihood.
• Figure 2: Out-of-distribution validation of the restorer guidance. The comparison methods are trained on Rain100L yang2017deep while evaluated on Rain100H yang2017deep.
• Figure 3: Visual comparison of restorer guidance with other inverse problem solvers on variational deterioration processes, including image dehazing, rain streak removal, and motion deblurring. The restorer prototype is deployed with NAFNet for comparison. Best viewed zoomed in.
• Figure 4: Visual results of out-of-distribution validation of the restorer guidance. First row: Trained on Rain100L and evaluated on Rain100H with PreNet prototype. Second row: Trained on GoPro and evaluated on RealBlur-J with Restormer prototype.