Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing

Liyang He, Zhenya Huang, Jiayu Liu, Enhong Chen, Fei Wang, Jing Sha, Shijin Wang

TL;DR

To ensure the effectiveness of two kinds of search paradigms in the context of semantic hashing, BRCD first aligns the semantic spaces between the teacher and student models through a contrastive knowledge distillation objective, and introduces a bit mask mechanism in the authors' knowledge distillation objective.

Abstract

Unsupervised semantic hashing has emerged as an indispensable technique for fast image search, which aims to convert images into binary hash codes without relying on labels. Recent advancements in the field demonstrate that employing large-scale backbones (e.g., ViT) in unsupervised semantic hashing models can yield substantial improvements. However, the inference delay has become increasingly difficult to overlook. Knowledge distillation provides a means for practical model compression to alleviate this delay. Nevertheless, the prevailing knowledge distillation approaches are not explicitly designed for semantic hashing. They ignore the unique search paradigm of semantic hashing, the inherent necessities of the distillation process, and the property of hash codes. In this paper, we propose an innovative Bit-mask Robust Contrastive knowledge Distillation (BRCD) method, specifically devised for the distillation of semantic hashing models. To ensure the effectiveness of two kinds of search paradigms in the context of semantic hashing, BRCD first aligns the semantic spaces between the teacher and student models through a contrastive knowledge distillation objective. Additionally, to eliminate noisy augmentations and ensure robust optimization, a cluster-based method within the knowledge distillation process is introduced. Furthermore, through a bit-level analysis, we uncover the presence of redundancy bits resulting from the bit independence property. To mitigate these effects, we introduce a bit mask mechanism in our knowledge distillation objective. Finally, extensive experiments not only showcase the noteworthy performance of our BRCD method in comparison to other knowledge distillation methods but also substantiate the generality of our methods across diverse semantic hashing models and backbones. The code for BRCD is available at https://github.com/hly1998/BRCD.

Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing

TL;DR

To ensure the effectiveness of two kinds of search paradigms in the context of semantic hashing, BRCD first aligns the semantic spaces between the teacher and student models through a contrastive knowledge distillation objective, and introduces a bit mask mechanism in the authors' knowledge distillation objective.

Abstract

Unsupervised semantic hashing has emerged as an indispensable technique for fast image search, which aims to convert images into binary hash codes without relying on labels. Recent advancements in the field demonstrate that employing large-scale backbones (e.g., ViT) in unsupervised semantic hashing models can yield substantial improvements. However, the inference delay has become increasingly difficult to overlook. Knowledge distillation provides a means for practical model compression to alleviate this delay. Nevertheless, the prevailing knowledge distillation approaches are not explicitly designed for semantic hashing. They ignore the unique search paradigm of semantic hashing, the inherent necessities of the distillation process, and the property of hash codes. In this paper, we propose an innovative Bit-mask Robust Contrastive knowledge Distillation (BRCD) method, specifically devised for the distillation of semantic hashing models. To ensure the effectiveness of two kinds of search paradigms in the context of semantic hashing, BRCD first aligns the semantic spaces between the teacher and student models through a contrastive knowledge distillation objective. Additionally, to eliminate noisy augmentations and ensure robust optimization, a cluster-based method within the knowledge distillation process is introduced. Furthermore, through a bit-level analysis, we uncover the presence of redundancy bits resulting from the bit independence property. To mitigate these effects, we introduce a bit mask mechanism in our knowledge distillation objective. Finally, extensive experiments not only showcase the noteworthy performance of our BRCD method in comparison to other knowledge distillation methods but also substantiate the generality of our methods across diverse semantic hashing models and backbones. The code for BRCD is available at https://github.com/hly1998/BRCD.
Paper Structure (31 sections, 28 equations, 6 figures, 3 tables)

This paper contains 31 sections, 28 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Hashing
  4. Knowledge Distillation
  5. PRELIMINARIES
  6. Semantic Hashing
  7. Semantic Hashing Search Paradigm with Knowledge Distillation
  8. The proposed BRCD method
  9. Contrastive Knowledge Distillation
  10. Cluster-based Robust Optimization
  11. Bit mask mechanism
  12. Experiments
  13. Knowledge Distillation Methods Comparison
  14. Setting
  15. Results
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Search paradigms for semantic hashing. (a) represents the Symmetric Semantic Hashing search Paradigm (SSHP), utilizing a single model for both offline and online stages. (b) depicts the Asymmetric Semantic Hashing search paradigm (ASHP), employing distinct models for each stage.
  • Figure 2: Workflow of BRCD. (a) The contrastive knowledge distillation achieves individual-space and structural-semantic knowledge transfer. (b) Clustering and assigning pseudo labels to images. (c) Cluster-based method eliminates offset positive and false negative samples. (d) The process to get bit masks. (e) Bit mask mechanism prevents incorrect optimization direction.
  • Figure 3: We conduct the analysis using 5000 images with the same class on the CIFAR-10 dataset. It shows the frequency histograms of eight randomly chosen dimensions.
  • Figure 4: The mAP@1000 results on the CIFAR-10 dataset when using different knowledge methods and backbones.
  • Figure 5: (a) The ISD and NRA@K on CIFAR-10 dataset for semantic space alignment analysis. (b) Hyper-parameters analysis on $\alpha$ and $\delta$.
  • ...and 1 more figures