Deep Graph Matching via Blackbox Differentiation of Combinatorial Solvers

TL;DR

This work presents BB-GM, an end to end trainable architecture for deep graph matching that embeds unmodified combinatorial graph matching solvers via blackbox differentiation. By combining strong visual and geometric features with a global feature gating mechanism and using implicit gradient through a solver, the method achieves state of the art on standard keypoint matching benchmarks and on SPair-71k, including novel unfiltered keypoint setups. The approach enables powerful post processing with multi graph matching and demonstrates robust performance across datasets with varying viewpoints and outliers, supported by comprehensive ablations. The results illustrate the practical potential of solver guided architectures for complex matching tasks and point to broader use in related combinatorial vision problems.

Abstract

Building on recent progress at the intersection of combinatorial optimization and deep learning, we propose an end-to-end trainable architecture for deep graph matching that contains unmodified combinatorial solvers. Using the presence of heavily optimized combinatorial solvers together with some improvements in architecture design, we advance state-of-the-art on deep graph matching benchmarks for keypoint correspondence. In addition, we highlight the conceptual advantages of incorporating solvers into deep learning architectures, such as the possibility of post-processing with a strong multi-graph matching solver or the indifference to changes in the training setting. Finally, we propose two new challenging experimental setups. The code is available at https://github.com/martius-lab/blackbox-deep-graph-matching

Figures (7)

• Figure 1: Example keypoint matchings of the proposed architecture on SPair-71k.
• Figure 2: Differentiation of a piecewise constant loss resulting from incorporating a graph matching solver. A two-dimensional section of the loss landscape is shown (left) along with two differentiable interpolations of increasing strengths (middle and right).
• Figure 3: Cycle consistency in multi-graph matching. The partial matching induced by light and dark green edges prohibits including the dashed edges.
• Figure 4: Extraction of features for a single image. Keypoint locations and VGG features are processed by a SplineCNN and a global feature vector is produced.
• Figure 5: Construction of combinatorial instance for keypoint matching.