Semantic-Aware Adversarial Training for Reliable Deep Hashing Retrieval

TL;DR

This paper conceive a discriminative mainstay features learning (DMFL) scheme to construct semantic representatives for guiding adversarial learning in deep hashing, and formulate the formalized adversarial training of deep hashing into a unified minimax optimization under the guidance of the generated mainstay codes.

Abstract

Deep hashing has been intensively studied and successfully applied in large-scale image retrieval systems due to its efficiency and effectiveness. Recent studies have recognized that the existence of adversarial examples poses a security threat to deep hashing models, that is, adversarial vulnerability. Notably, it is challenging to efficiently distill reliable semantic representatives for deep hashing to guide adversarial learning, and thereby it hinders the enhancement of adversarial robustness of deep hashing-based retrieval models. Moreover, current researches on adversarial training for deep hashing are hard to be formalized into a unified minimax structure. In this paper, we explore Semantic-Aware Adversarial Training (SAAT) for improving the adversarial robustness of deep hashing models. Specifically, we conceive a discriminative mainstay features learning (DMFL) scheme to construct semantic representatives for guiding adversarial learning in deep hashing. Particularly, our DMFL with the strict theoretical guarantee is adaptively optimized in a discriminative learning manner, where both discriminative and semantic properties are jointly considered. Moreover, adversarial examples are fabricated by maximizing the Hamming distance between the hash codes of adversarial samples and mainstay features, the efficacy of which is validated in the adversarial attack trials. Further, we, for the first time, formulate the formalized adversarial training of deep hashing into a unified minimax optimization under the guidance of the generated mainstay codes. Extensive experiments on benchmark datasets show superb attack performance against the state-of-the-art algorithms, meanwhile, the proposed adversarial training can effectively eliminate adversarial perturbations for trustworthy deep hashing-based retrieval. Our code is available at https://github.com/xandery-geek/SAAT.

Figures (7)

• Figure 1: The comparison between classification and semantic similarity-preserving hashing on adversarial learning. In the output space of classification, an adversarial attack with the guidance of labels only needs to make the adversarial samples across the decision boundary. However, an adversarial attack on deep hashing is more challenging because the adversarial samples are expected to blend into the clusters in the embedding space, but there is not an explicit signal to supervise the process. The same problem exists for adversarial training. For classification, adversarial training just maximizes probabilities of adversarial samples on true labels, which is infeasible on hashing due to the lack of explicit supervision signals. Hence, deep hashing needs reliable and discriminative semantic representatives (i.e., mainstay) to represent the image semantics for adversarial attacks and defense.
• Figure 2: The pipeline of the proposed Semantic-Aware Adversarial Training (SAAT) for deep hashing retrieval. The architecture is composed of two mechanisms: the generation branch of representative codes (i.e., mainstay codes) and the minimax-based adversarial training branch. In mainstay code generation, all training samples are projected to the Hamming space to form their corresponding hash codes. Then, we build a mainstay code for each class by discriminative learning which pulls the mainstay code closer to the hash code of positives as well as pushes it away from other negatives. In adversarial training, we extend adversarial training of deep hashing to a minimax framework i.e., standard adversarial training. As illustrated, the gray, red, and blue arrows indicate forward, backward and forward propagation, respectively. The red arrow means constructing adversarial samples with the supervision of generated mainstay codes and the blue arrow represents inputting adversarial samples for adversarial training. Best viewed in color.
• Figure 3: Precision-Recall curves on FLICKR-25K and MS-COCO under 32 bits code length.
• Figure 4: Precision@1000 curves on FLICKR-25K and MS-COCO under 32 bits code length.
• Figure 5: Retrieval examples on NUS-WIDE with the benign query and its adversarial version. We provide examples of non-targeted attacks and targeted attacks in (a) and (b), respectively. For each example, the two boxes represent the top-5 retrieved images of the natural and adversarial queries, respectively.