Gromov Wasserstein Optimal Transport for Semantic Correspondences

TL;DR

This work rethinks semantic correspondences by embedding spatial structure directly into a matching algorithm through Gromov-Wasserstein optimal transport. By adopting a partially unbalanced GW formulation and a symmetry-aware term, the method achieves spatially coherent, unique correspondences using only DINOv2 features, delivering competitive or superior results with much lower computational cost than diffusion-based baselines. Ablations show the critical contributions of the GW prior, unbalanced mass handling, and symmetry constraints. The approach yields state-of-the-art performance on the TSS dataset and strong results on PF-PASCAL and SPair-71k, highlighting its practical impact for robust, efficient semantic matching in vision tasks.

Abstract

Establishing correspondences between image pairs is a long studied problem in computer vision. With recent large-scale foundation models showing strong zero-shot performance on downstream tasks including classification and segmentation, there has been interest in using the internal feature maps of these models for the semantic correspondence task. Recent works observe that features from DINOv2 and Stable Diffusion (SD) are complementary, the former producing accurate but sparse correspondences, while the latter produces spatially consistent correspondences. As a result, current state-of-the-art methods for semantic correspondence involve combining features from both models in an ensemble. While the performance of these methods is impressive, they are computationally expensive, requiring evaluating feature maps from large-scale foundation models. In this work we take a different approach, instead replacing SD features with a superior matching algorithm which is imbued with the desirable spatial consistency property. Specifically, we replace the standard nearest neighbours matching with an optimal transport algorithm that includes a Gromov Wasserstein spatial smoothness prior. We show that we can significantly boost the performance of the DINOv2 baseline, and be competitive and sometimes surpassing state-of-the-art methods using Stable Diffusion features, while being 5--10x more efficient. We make code available at https://github.com/fsnelgar/semantic_matching_gwot .

Figures (8)

• Figure 1: Accuracy and latency trade off for various methods. Our method achieves comparable accuracy while being much faster and requiring less memory. The size of the markers indicates the relative number of parameters in each method, and the colour indicates the model family. Green methods use DINO models, orange methods use Stable Diffusion, and red methods combine features from both. Methods marked with $\dagger$ use ground truth labels to flip keypoints at test time. See \ref{['tab:comp_results']} for a detailed breakdown.
• Figure 2: Impact of Gromov Wasserstein optimal transport. Each two-by-two grid contains an example image pair on the left, and dense correspondences using nearest neighbours and our GW method in the top right and bottom right respectively. Note that correspondences for our method are more spatially consistent.
• Figure 3: Effect of the symmetry loss. Each two-by-two grid contains matches without the symmetry loss on the left, and with the symmetry loss on the right. Note that with the symmetry loss ordering of the keypoint pair is maintained and matches are more plausible.
• Figure 4: Impact of unbalanced optimal transport. Each two-by-two grid shows the image pair on the left, with results for balanced and partially balanced optimal transport on the top right and bottom right respectively. For objects with large scale differences (first and fourth examples) or occlusion (second and third examples), pixels have unequal importance and the balanced mass assumption doesn't hold. The unbalanced OT results are more plausible.
• Figure 5: Failure cases of our method. Each two-by-two grid shows dense correspondences for a failure case. First and third examples show failure due to scale, second shows failure due to incorrect GW smoothing, and fourth failure due to invalid symmetry assumption.