Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Descriptor Distillation: a Teacher-Student-Regularized Framework for Learning Local Descriptors

Yuzhen Liu, Qiulei Dong

TL;DR

Experimental results on 3 public datasets demonstrate that the equal-weight student models, derived from the proposed DesDis framework by utilizing three typical descriptor learning networks as teacher models, could achieve significantly better performances than their teachers and several other comparative methods.

Abstract

Learning a fast and discriminative patch descriptor is a challenging topic in computer vision. Recently, many existing works focus on training various descriptor learning networks by minimizing a triplet loss (or its variants), which is expected to decrease the distance between each positive pair and increase the distance between each negative pair. However, such an expectation has to be lowered due to the non-perfect convergence of network optimizer to a local solution. Addressing this problem and the open computational speed problem, we propose a Descriptor Distillation framework for local descriptor learning, called DesDis, where a student model gains knowledge from a pre-trained teacher model, and it is further enhanced via a designed teacher-student regularizer. This teacher-student regularizer is to constrain the difference between the positive (also negative) pair similarity from the teacher model and that from the student model, and we theoretically prove that a more effective student model could be trained by minimizing a weighted combination of the triplet loss and this regularizer, than its teacher which is trained by minimizing the triplet loss singly. Under the proposed DesDis, many existing descriptor networks could be embedded as the teacher model, and accordingly, both equal-weight and light-weight student models could be derived, which outperform their teacher in either accuracy or speed. Experimental results on 3 public datasets demonstrate that the equal-weight student models, derived from the proposed DesDis framework by utilizing three typical descriptor learning networks as teacher models, could achieve significantly better performances than their teachers and several other comparative methods. In addition, the derived light-weight models could achieve 8 times or even faster speeds than the comparative methods under similar patch verification performances

Descriptor Distillation: a Teacher-Student-Regularized Framework for Learning Local Descriptors

TL;DR

Experimental results on 3 public datasets demonstrate that the equal-weight student models, derived from the proposed DesDis framework by utilizing three typical descriptor learning networks as teacher models, could achieve significantly better performances than their teachers and several other comparative methods.

Abstract

Learning a fast and discriminative patch descriptor is a challenging topic in computer vision. Recently, many existing works focus on training various descriptor learning networks by minimizing a triplet loss (or its variants), which is expected to decrease the distance between each positive pair and increase the distance between each negative pair. However, such an expectation has to be lowered due to the non-perfect convergence of network optimizer to a local solution. Addressing this problem and the open computational speed problem, we propose a Descriptor Distillation framework for local descriptor learning, called DesDis, where a student model gains knowledge from a pre-trained teacher model, and it is further enhanced via a designed teacher-student regularizer. This teacher-student regularizer is to constrain the difference between the positive (also negative) pair similarity from the teacher model and that from the student model, and we theoretically prove that a more effective student model could be trained by minimizing a weighted combination of the triplet loss and this regularizer, than its teacher which is trained by minimizing the triplet loss singly. Under the proposed DesDis, many existing descriptor networks could be embedded as the teacher model, and accordingly, both equal-weight and light-weight student models could be derived, which outperform their teacher in either accuracy or speed. Experimental results on 3 public datasets demonstrate that the equal-weight student models, derived from the proposed DesDis framework by utilizing three typical descriptor learning networks as teacher models, could achieve significantly better performances than their teachers and several other comparative methods. In addition, the derived light-weight models could achieve 8 times or even faster speeds than the comparative methods under similar patch verification performances
Paper Structure (31 sections, 1 theorem, 9 equations, 9 figures, 8 tables)

This paper contains 31 sections, 1 theorem, 9 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DNN-based Local Descriptors
  4. Knowledge Distillation in Other Visual Tasks
  5. Methodology
  6. The DesDis Framework
  7. Teacher-Student Regularizer
  8. Total Loss Function
  9. Theoretical Analysis
  10. Experimental Results
  11. Datasets and Comparison Group
  12. Implementation Details
  13. Equal-Weight Student Models Under DesDis
  14. Evaluation on Brown
  15. Evaluation on HPatches
  16. ...and 16 more sections

Key Result

Proposition 1

Given the distances $\{d_i^{\mathrm{t},+}\}_{i=1}^N$ of N positive descriptor pairs and the distances $\{d_i^{\mathrm{t},-}\}_{i=1}^N$ of N negative descriptor pairs in the teacher model, under the condition that $m+d_i^{\mathrm{s},+}-d_i^{\mathrm{s},-}>0,i=1,2,\cdots,N$, the optimal solution to the

Figures (9)

  • Figure 1: The pipeline of the proposed DesDis framework. $d^{\mathrm{t},+}$ and $d^{\mathrm{t},-}$ are the distances of positive and negative descriptor pairs from the teacher model. $d^{\mathrm{s},+}$ and $d^{\mathrm{s},-}$ are the distances of positive and negative descriptor pairs from the student model. $\mathcal{L}_\mathrm{B}$ is a triplet loss variant. $\mathcal{L}_\mathrm{TSP}$ and $\mathcal{L}_\mathrm{TSN}$ are the two forms of the teacher-student regularizer.
  • Figure 2: Architectures of the design light-weight model DesDis-$D$ which consists of 5 convolutional layers. '$\backslash$2' denotes strided convolution with a stride of 2.
  • Figure 3: The distribution of (a) '$m+d^{\mathrm{s},+}-d^{\mathrm{s},-}$' from DesDis-HardNet on Brown, (b) '$m+d^{\mathrm{s},+}-d^{\mathrm{s},-}$' from DesDis-HardNet on HPatches, (c) '$m+d^{\mathrm{s},+}-d^{\mathrm{s},-}$' from DesDis-32 (using SIFT as teacher) on Brown, and (d) '$m+d^{\mathrm{s},+}-d^{\mathrm{s},-}$' from DesDis-32 (using SIFT as teacher) on HPatches. '$d^{\mathrm{s},+}$' and '$d^{\mathrm{s},-}$' denote the distance of positive and negative pairs respectively from the student.
  • Figure 4: Comparison on the test set of split 'a' of the HPatches dataset HPatches. (a) Evaluation on the derived equal-weight student models and several typical methods SIFTTNetHardNetSOSNetHyNet. 'DesDis-Hard', 'DesDis-SOS' and 'DesDis-Hy' denote DesDis-HardNet, DesDis-SOSNet and DesDis-HyNet respectively. (b) Evaluation on the derived light-weight student models trained with/without the proposed teacher-student regularizer. '$\dagger$' denotes the baseline models that are trained without the proposed teacher-student regularizer.
  • Figure 5: Image matching visualization on HPatches by the proposed DesDis-32, HyNet, TFeat and SIFT. The models are trained on the Liberty subset of Brown. The number of correct matches is shown in the upper left corner of each image pair.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Proposition 1
  • proof