Unified Negative Pair Generation toward Well-discriminative Feature Space for Face Recognition

TL;DR

Framing face recognition as a pair-similarity optimization problem, the paper identifies a mismatch between sampled negative pairs and the full negative distribution that hinders forming a well-discriminative feature space. It introduces Unified Negative Pair Generation (UNPG), which unifies metric-based PG (MLPG) and classification-based PG (CLPG) and adds noise-negative pair filtering to stabilize training. A unified loss $\mathcal{L}^{uni}$ incorporates positive and negative scores from both sources, enabling better separation in the feature space and approaching the ideal condition $\inf{\mathcal{S}^p} > \sup{\mathcal{S}^n}$. Extensive experiments on IJB-B, IJB-C, MegaFace, and K-FACE show consistent gains across ArcFace, CosFace, and MagFace variants, with code and pretrained models publicly available.

Abstract

The goal of face recognition (FR) can be viewed as a pair similarity optimization problem, maximizing a similarity set $\mathcal{S}^p$ over positive pairs, while minimizing similarity set $\mathcal{S}^n$ over negative pairs. Ideally, it is expected that FR models form a well-discriminative feature space (WDFS) that satisfies $\inf{\mathcal{S}^p} > \sup{\mathcal{S}^n}$. With regard to WDFS, the existing deep feature learning paradigms (i.e., metric and classification losses) can be expressed as a unified perspective on different pair generation (PG) strategies. Unfortunately, in the metric loss (ML), it is infeasible to generate negative pairs taking all classes into account in each iteration because of the limited mini-batch size. In contrast, in classification loss (CL), it is difficult to generate extremely hard negative pairs owing to the convergence of the class weight vectors to their center. This leads to a mismatch between the two similarity distributions of the sampled pairs and all negative pairs. Thus, this paper proposes a unified negative pair generation (UNPG) by combining two PG strategies (i.e., MLPG and CLPG) from a unified perspective to alleviate the mismatch. UNPG introduces useful information about negative pairs using MLPG to overcome the CLPG deficiency. Moreover, it includes filtering the similarities of noisy negative pairs to guarantee reliable convergence and improved performance. Exhaustive experiments show the superiority of UNPG by achieving state-of-the-art performance across recent loss functions on public benchmark datasets. Our code and pretrained models are publicly available.

Figures (6)

• Figure 1: Similarity distributions viewed from pair generation perspective for approximating WDFS. The bottom line presents similarity distributions in feature space after sufficiently learning in their own ways with the top line. $\mathcal{S}^p$ and $\mathcal{S}^n$ are positive and negative similarity sets and $\mathcal{\hat{S}}^p$ and $\mathcal{\hat{S}}^n$ are subsets of $\mathcal{S}^p$ and $\mathcal{S}^n$, respectively. (a) The ideal similarity sets satisfying $\inf{\mathcal{S}^p} > \sup{\mathcal{S}^n}$ after learning with $\mathcal{S}^p$ and $\mathcal{S}^n$. $\theta^{p}_{max}$ and $\theta^{n}_{min}$ are the max and min angles among positive and negative pairs. (b) Using a vanilla loss, no overlap between $\mathcal{\hat{S}}^p$ and $\mathcal{\hat{S}}^n$ results in an overlap between $\mathcal{S}^p$ and $\mathcal{S}^n$. (c) Using a marginal loss, an overlap between $\mathcal{\hat{S}}^p$ and $\mathcal{\hat{S}}^n$ by shifting $\mathcal{\hat{S}}^p$ reduces an overlap after learning. (d) Using more negative pairs, an overlap between $\mathcal{\hat{S}}^p$ and $\mathcal{\hat{S}}^n$ by shifting $\mathcal{\hat{S}}^p$ and enlarging $\mathcal{\hat{S}}^n$ significantly reduces the overlap after learning.
• Figure 2: Unified loss with UNPG.
• Figure 3: Geometrical interpretation of feature space associated with similarity space. (a) As ideal behavior of the loss function, it imposes a large loss in feature space with low discriminability. A shading area in the same color represents the target region of the same class. $\theta^p_{max}$ and $\theta^n_{min}$are the respective angles of max positive and min negative pairs in the feature space. $\mathcal{S}^p$ and $\mathcal{S}^n$ represent similarity sets. (b) In spite of being equally low discriminative, a very small loss is given by vanilla loss (e.g., norm-softmax). $\bm{w}_1$ and $\bm{w}_2$ are the normalized weight vectors of classes 1 and 2, while $\bm{x}_1$ and $\bm{x}_2$ are the normalized feature vector. $\hat{\theta}^p_{max}$ and $\hat{\theta}^n_{min}$ represent the angle of max positive and min negative pairs in $\hat{\mathcal{S}}^p \subset \mathcal{S}^p$ and $\hat{\mathcal{S}}^n \subset \mathcal{S}^n$, respectively. (c) Mismatch between $\mathcal{S}^p$ and $\mathcal{\hat{S}}^p$ is reduced by using a marinal classification loss (e.g., ArcFace). However, still a small loss is given because of a mismatch between $\mathcal{S}^n$ and $\mathcal{\hat{S}}^n$. (d) Marginal classification loss with UNPG behaves closest to ideal by alleviating mismatch between $\mathcal{S}^n$ and $\mathcal{\hat{S}}^n$.
• Figure 4: Comparison of overlapping similarities for positive and negative pairs with and without UNPG.
• Figure 5: (a) Effects of noise negative pair filtering in UNPG with ResNet-34 (b) Examples of three negative pair types in ascending order of similarity scores.