Diffusion Model for Dense Matching

TL;DR

DiffMatch reframes dense pixel-wise image matching as learning a posterior over the matching field by using a conditional diffusion model. It jointly models the data term and a learned prior through a conditional denoising diffusion process, augmented by a cost-injection pathway with an initial flow and local costs, and a cascaded flow-upsampling scheme to recover high-resolution correspondences. Through ablations and extensive experiments on HPatches, ETH3D, and corrupted benchmarks, the method achieves state-of-the-art results and demonstrates robustness to challenging textures, repetitions, large displacements, and noise. The work highlights the value of explicitly incorporating a generative prior for dense matching and provides practical inference strategies to harness diffusion models for discriminative tasks.

Abstract

The objective for establishing dense correspondence between paired images consists of two terms: a data term and a prior term. While conventional techniques focused on defining hand-designed prior terms, which are difficult to formulate, recent approaches have focused on learning the data term with deep neural networks without explicitly modeling the prior, assuming that the model itself has the capacity to learn an optimal prior from a large-scale dataset. The performance improvement was obvious, however, they often fail to address inherent ambiguities of matching, such as textureless regions, repetitive patterns, and large displacements. To address this, we propose DiffMatch, a novel conditional diffusion-based framework designed to explicitly model both the data and prior terms. Unlike previous approaches, this is accomplished by leveraging a conditional denoising diffusion model. DiffMatch consists of two main components: conditional denoising diffusion module and cost injection module. We stabilize the training process and reduce memory usage with a stage-wise training strategy. Furthermore, to boost performance, we introduce an inference technique that finds a better path to the accurate matching field. Our experimental results demonstrate significant performance improvements of our method over existing approaches, and the ablation studies validate our design choices along with the effectiveness of each component. Project page is available at https://ku-cvlab.github.io/DiffMatch/.

Figures (12)

• Figure 1: Visualizing the effectiveness of the proposed DiffMatch: (a) source images, (b) target images, and warped source images using estimated correspondences by (c-d) state-of-the-art approaches truong2020glutruong2020gocor, (e) our DiffMatch, and (f) ground-truth. Compared to previous methods truong2020glutruong2020gocor that discriminatively estimate correspondences, our diffusion-based generative framework effectively learns the matching field manifold, resulting in better estimating correspondences particularly at textureless regions, repetitive patterns, and large displacements.
• Figure 2: Overall network architecture of DiffMatch. Given source and target images, our conditional diffusion-based network estimates the dense correspondence between the two images. We leverage two conditions: the initial correspondence $F_\mathrm{init}$ and the local matching cost $C^l$, which finds long-range matching and embeds local pixel-wise interactions, respectively.
• Figure 3: Visualization of the reverse diffusion process in DiffMatch: (from left to right) source and target images, and warped source images by estimated correspondences as evolving time steps. The source image is progressively warped into the target image through an iterative denoising process.
• Figure 4: Qualitative results on HPatches balntas2017hpatches using motion blur in hendrycks2019benchmarking. The source images are warped to the target images using predicted correspondences.
• Figure 5: Qualitative results on HPatches balntas2017hpatches. the source images are warped to the target images using predicted correspondences.