Globally Correlation-Aware Hard Negative Generation

TL;DR

A globally correlation-aware hard negative generation (GCA-HNG) framework, which first learns sample correlations from a global perspective and exploits these correlations to guide generating hardness-adaptive and diverse negatives.

Abstract

Hard negative generation aims to generate informative negative samples that help to determine the decision boundaries and thus facilitate advancing deep metric learning. Current works select pair/triplet samples, learn their correlations, and fuse them to generate hard negatives. However, these works merely consider the local correlations of selected samples, ignoring global sample correlations that would provide more significant information to generate more informative negatives. In this work, we propose a Globally Correlation-Aware Hard Negative Generation (GCA-HNG) framework, which first learns sample correlations from a global perspective and exploits these correlations to guide generating hardness-adaptive and diverse negatives. Specifically, this approach begins by constructing a structured graph to model sample correlations, where each node represents a specific sample and each edge represents the correlations between corresponding samples. Then, we introduce an iterative graph message propagation to propagate the messages of node and edge through the whole graph and thus learn the sample correlations globally. Finally, with the guidance of the learned global correlations, we propose a channel-adaptive manner to combine an anchor and multiple negatives for HNG. Compared to current methods, GCA-HNG allows perceiving sample correlations with numerous negatives from a global and comprehensive perspective and generates the negatives with better hardness and diversity. Extensive experiment results demonstrate that the proposed GCA-HNG is superior to related methods on four image retrieval benchmark datasets. Codes and trained models are available at \url{https://github.com/PWenJay/GCA-HNG}.

Figures (8)

• Figure 1: The t-SNE visualization depicts the embedding distributions of four classes on the Cars196 dataset, with each class represented by a distinct colored shape. This illustration emphasizes the importance of considering global correlations with negatives from diverse classes when analyzing a specific anchor for HNG. It points out the necessity of perceiving the global geometric distribution to generate harder synthetic negatives for closely related classes, thereby enhancing class discrimination. In contrast, for more distantly related negatives, it should control the hardness of the synthetic negative reasonably to avoid deviating from its corresponding class distribution.
• Figure 2: Schematic of the GCA-HNG framework. We use an embedding representation $\textbf{z}_1$ as the anchor and the embedding representations $\textbf{z}_3 \sim \textbf{z}_6$ as negatives to generate the hard negatives of $\textbf{z}_1$ as an illustrative example for clarity. GCA-HNG consists of a GCL module and a CACAI module. The former constructs a graph for representations $\textbf{z}$ and introduces an iterative graph message propagation mechanism, where node message propagation implements node-to-node and edge-to-node interaction to perceive global correlations, and edge message propagation implements node-to-edge interaction to facilitate the edges to model the correlations between adjacent samples globally. The nodes and edges are updated iteratively. The latter uses the learned global correlations $E_{1\cdot}^{K}$ to produce channel-adaptive interpolation vectors $\bm{\lambda}_{1\cdot}$ for each anchor-negative pair and integrates the anchor $\textbf{z}_1$ with multiple negatives from a specific class (e.g., $\textbf{z}_3,\textbf{z}_4$) by random weighting (RW) to generate the corresponding informative negatives (e.g., $\hat{\textbf{z}}_{12}$) of the anchor $\textbf{z}_1$.
• Figure 3: Implementation details diagram of the proposed iterative graph message propagation mechanism, where blue arrows indicate node message propagation routes and green arrows indicate edge message propagation routes.
• Figure 4: Comparison of embedding representation distributions using t-SNE visualization for the CUB-200-2011 and Cars196 datasets after the same training iterations for HDML and our proposed GCA-HNG. The red circles present numerous noisy samples generated by HDML that disturb embedding space optimization. Our proposed GCA-HNG adaptively adjusts the hardness of the synthetic negatives to enhance their informativeness, significantly improving the convergence effect in the metric learning process.
• Figure 5: Comparison of retrieval results between the Baseline and GCA-HNG for four images on the CUB-200-2011, Cars196, SOP, and InShop datasets. The image to the left of the dashed line is the query, while the images to the right are Top-4 retrieval results, respectively. The green box indicates that the retrieved result belongs to the same category as the query, and the red box vice versa.