Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentiable Proximal Graph Matching

Haoru Tan, Chuang Wang, Xu-Yao Zhang, Cheng-Lin Liu

TL;DR

This work tackles graph matching in vision by reframing the NP-hard quadratic assignment problem into a differentiable, entropy-regularized relaxation. The Differentiable Proximal Graph Matching (DPGM) framework uses a proximal gradient approach with a KL-based proximal term and Sinkhorn normalization to produce a differentiable map from the affinity matrix to node correspondences. The affinity M is decomposed into unary and pairwise terms, which can be data-driven and learned within an end-to-end network, and convergence to a stationary point is established under mild conditions. DPGM is demonstrated to outperform existing graph matching methods on synthetic data, CMU House sequences, and PASCAL VOC keypoints, especially when integrated with deep feature extractors, underscoring the practical impact of incorporating a differentiable GM layer into deep learning pipelines.

Abstract

Graph matching is a fundamental tool in computer vision and pattern recognition. In this paper, we introduce an algorithm for graph matching based on the proximal operator, referred to as differentiable proximal graph matching (DPGM). Specifically, we relax and decompose the quadratic assignment problem for the graph matching into a sequence of convex optimization problems. The whole algorithm can be considered as a differentiable map from the graph affinity matrix to the prediction of node correspondence. Therefore, the proposed method can be organically integrated into an end-to-end deep learning framework to jointly learn both the deep feature representation and the graph affinity matrix. In addition, we provide a theoretical guarantee to ensure the proposed method converges to a stable point with a reasonable number of iterations. Numerical experiments show that PGM outperforms existing graph matching algorithms on diverse datasets such as synthetic data, and CMU House. Meanwhile, PGM can fully harness the capability of deep feature extractors and achieve state-of-art performance on PASCAL VOC keypoints.

Differentiable Proximal Graph Matching

TL;DR

This work tackles graph matching in vision by reframing the NP-hard quadratic assignment problem into a differentiable, entropy-regularized relaxation. The Differentiable Proximal Graph Matching (DPGM) framework uses a proximal gradient approach with a KL-based proximal term and Sinkhorn normalization to produce a differentiable map from the affinity matrix to node correspondences. The affinity M is decomposed into unary and pairwise terms, which can be data-driven and learned within an end-to-end network, and convergence to a stationary point is established under mild conditions. DPGM is demonstrated to outperform existing graph matching methods on synthetic data, CMU House sequences, and PASCAL VOC keypoints, especially when integrated with deep feature extractors, underscoring the practical impact of incorporating a differentiable GM layer into deep learning pipelines.

Abstract

Graph matching is a fundamental tool in computer vision and pattern recognition. In this paper, we introduce an algorithm for graph matching based on the proximal operator, referred to as differentiable proximal graph matching (DPGM). Specifically, we relax and decompose the quadratic assignment problem for the graph matching into a sequence of convex optimization problems. The whole algorithm can be considered as a differentiable map from the graph affinity matrix to the prediction of node correspondence. Therefore, the proposed method can be organically integrated into an end-to-end deep learning framework to jointly learn both the deep feature representation and the graph affinity matrix. In addition, we provide a theoretical guarantee to ensure the proposed method converges to a stable point with a reasonable number of iterations. Numerical experiments show that PGM outperforms existing graph matching algorithms on diverse datasets such as synthetic data, and CMU House. Meanwhile, PGM can fully harness the capability of deep feature extractors and achieve state-of-art performance on PASCAL VOC keypoints.
Paper Structure (28 sections, 2 theorems, 20 equations, 2 figures, 1 table)

This paper contains 28 sections, 2 theorems, 20 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Objective.
  4. Formulation.
  5. Decomposition and Construction of the Affinity Matrix $\boldsymbol{M}$.
  6. Proximal Graph Matching
  7. Continuous relaxation and entropy regulerizer
  8. Proximal method
  9. Introduction of the proximal method.
  10. Proximal Graph Matching
  11. Convergence Analysis
  12. Integration of DPGM into Deep Neural Networks
  13. Problem Setting of key points matching
  14. Feature extractor module
  15. Optional Graph Convolution Module
  16. ...and 13 more sections

Key Result

lemma thmcounterlemma

There exist a constant $\alpha >0$ such that for all $\boldsymbol{z}_{t+1}$, $\boldsymbol{z}_t$ generated by eq:prox-GM, we have where $D$ is the KL-divergence defined in eq:KL.

Figures (2)

  • Figure 1: Comparison of the matching accuracy, robustness of DPGM and other baselines on synthetic graph dataset.
  • Figure 2: Comparison of the matching accuracy, robustness of PGM and other baselines on CMU house sequence dataset.

Theorems & Definitions (4)

  • remark thmcounterremark
  • remark thmcounterremark
  • lemma thmcounterlemma
  • proposition thmcounterproposition