EPL: Empirical Prototype Learning for Deep Face Recognition

TL;DR

This work introduces Empirical Prototype Learning (EPL) for deep face recognition, addressing the drift of class prototypes caused by hard samples in traditional prototype-learning frameworks. EPL defines class prototypes as empirical expectations of per-class features and maintains adaptive empirical prototypes $P_i^{(e)}$ updated from positive samples via $P_i^{(e)}=\alpha_x P_i^{(e)}+(1-\alpha_x)x$ with $\alpha_x=\sigma(s(x, P_i^{(e)}))$, while applying an adaptive-margin loss $L_{ePr-m}$ that uses $m_{x^{(i)}}=\text{Detach}(\tfrac{1}{\tau}s(x^{(i)}, P_i^{(e)}))$ (with $\tau=1/64$) to balance normal and hard samples. The method combines empirical-prototype and conventional-prototype terms into a unified objective, enabling hard samples to be pulled toward empirical centers without destabilizing class representations. Extensive experiments on CASIA-WebFace, WebFace4M/12M, Glint360K, and multiple benchmarks (MFR Ongoing and MegaFace Challenge 1) show consistent improvements over CosFace, ArcFace, UniFace, and related hybrids, especially on $1:\!n$ identification and challenging subsets like the Mask and IJB-C, while EPL maintains compatibility with existing prototype-learning losses. The results suggest EPL provides a practical, scalable way to strengthen prototype-based FR by leveraging the aggregate information of normal samples to better reflect class centers and relegate hard-sample influence to the guidance of empirical prototypes.

Abstract

Prototype learning is widely used in face recognition, which takes the row vectors of coefficient matrix in the last linear layer of the feature extraction model as the prototypes for each class. When the prototypes are updated using the facial sample feature gradients in the model training, they are prone to being pulled away from the class center by the hard samples, resulting in decreased overall model performance. In this paper, we explicitly define prototypes as the expectations of sample features in each class and design the empirical prototypes using the existing samples in the dataset. We then devise a strategy to adaptively update these empirical prototypes during the model training based on the similarity between the sample features and the empirical prototypes. Furthermore, we propose an empirical prototype learning (EPL) method, which utilizes an adaptive margin parameter with respect to sample features. EPL assigns larger margins to the normal samples and smaller margins to the hard samples, allowing the learned empirical prototypes to better reflect the class center dominated by the normal samples and finally pull the hard samples towards the empirical prototypes through the learning. The extensive experiments on MFR, IJB-C, LFW, CFP-FP, AgeDB, and MegaFace demonstrate the effectiveness of EPL. Our code is available at $\href{https://github.com/WakingHours-GitHub/EPL}{https://github.com/WakingHours-GitHub/EPL}$.

Figures (2)

• Figure 1: The pipeline of Empirical Prototype Learning (EPL) for deep face recognition. For a batch containing $B$ samples, $[X_k^{(i_k)}]_{k=1}^B$, the encoder extract the features, $[x_k^{(i_k)}]_{k=1}^B\in\mathbb R^{d\times B}$. Then, the adaptive coefficients $\alpha_x$ are calculated to update the empirical prototype corresponding to the current mini-batch of samples $P^{(\text{e})}_{i_k}$. Finally, the similarities, $s(x_k^{i_k}, P_j)$ and $s(x_k^{i_k}, P_j^{\text{(e)}})$, are used to calculate the loss $L_{\text{ePr-m}}$ for the encoder training.
• Figure 2: Similarity distribution between samples and different prototypes.