Constrained Particle Seeking: Solving Diffusion Inverse Problems with Just Forward Passes

TL;DR

A novel gradient-free approach that leverages all candidate particle information to actively search for the optimal particle while incorporating constraints aligned with high-density regions of the unconditional prior, enabling more flexible and efficient particle seeking.

Abstract

Diffusion models have gained prominence as powerful generative tools for solving inverse problems due to their ability to model complex data distributions. However, existing methods typically rely on complete knowledge of the forward observation process to compute gradients for guided sampling, limiting their applicability in scenarios where such information is unavailable. In this work, we introduce \textbf{\emph{Constrained Particle Seeking (CPS)}}, a novel gradient-free approach that leverages all candidate particle information to actively search for the optimal particle while incorporating constraints aligned with high-density regions of the unconditional prior. Unlike previous methods that passively select promising candidates, CPS reformulates the inverse problem as a constrained optimization task, enabling more flexible and efficient particle seeking. We demonstrate that CPS can effectively solve both image and scientific inverse problems, achieving results comparable to gradient-based methods while significantly outperforming gradient-free alternatives. Code is available at https://github.com/deng-ai-lab/CPS.

Figures (9)

• Figure 1: Qualitative comparison between CPS and baseline methods. The proposed CPS relies solely on the forward pass of the observation model to solve both image and scientific inverse problems. It performs competitively with advanced gradient-based methods and significantly outperforms gradient-free methods. The best results are highlighted in red, and the second-best results are shown in blue.
• Figure 2: Empirical results of black hole imaging for different particle selections in SCG. The top shows the results obtained by selecting the first few particles in the ranking, while the bottom demonstrates the effect of selecting the last few particles and reversing the sign of their noise.
• Figure 3: Overview of Constrained Particle Seeking (CPS) for inverse problem solving. At each time step, the process involves three steps. Step 1: Sample multiple candidate particles; Step 2: Fit the forward process surrogate using these samples; Step 3: Use the surrogate to determine the optimal particle on the hypersphere, as the next state.
• Figure 4: Evolution of Jensen gap of CPS and EnKG for 4$\times$ super-resolution on the FFHQ dataset.
• Figure 5: Qualitative results of image inverse problems on FFHQ: from top to bottom, inpainting, super-resolution, deblurring, and JPEG restoration.