GIF: Generative Inspiration for Face Recognition at Scale

TL;DR

This work tackles the prohibitive cost of Softmax in large-scale face recognition by replacing scalar class labels with structured identity codes that are tokenized on a hyperspherical code space. The method, GIF, consists of two stages: constructing identity codes via hyperspherical code vectors initialized with CLIP (and regularized for uniform distribution) and hierarchical clustering, then training an FR backbone to predict tokenized codes rather than class labels, yielding a logarithmic scaling of computation with respect to the number of identities. The approach includes a joint objective with cross-entropy over token predictions and an intra-class alignment term, and demonstrates substantial memory and speed benefits while achieving state-of-the-art accuracy on multiple benchmarks (e.g., TAR@FAR$=10^{-4}$ on IJB-B/C) across backbones. The results indicate that structured identity codes mitigate minority collapse, improve inter-class separability, and enable scalable FR training without sacrificing performance, making large-scale FR more practical for real-world deployment.

Abstract

Aiming to reduce the computational cost of Softmax in massive label space of Face Recognition (FR) benchmarks, recent studies estimate the output using a subset of identities. Although promising, the association between the computation cost and the number of identities in the dataset remains linear only with a reduced ratio. A shared characteristic among available FR methods is the employment of atomic scalar labels during training. Consequently, the input to label matching is through a dot product between the feature vector of the input and the Softmax centroids. Inspired by generative modeling, we present a simple yet effective method that substitutes scalar labels with structured identity code, i.e., a sequence of integers. Specifically, we propose a tokenization scheme that transforms atomic scalar labels into structured identity codes. Then, we train an FR backbone to predict the code for each input instead of its scalar label. As a result, the associated computational cost becomes logarithmic w.r.t. number of identities. We demonstrate the benefits of the proposed method by conducting experiments. In particular, our method outperforms its competitors by 1.52%, and 0.6% at TAR@FAR$=1e-4$ on IJB-B and IJB-C, respectively, while transforming the association between computational cost and the number of identities from linear to logarithmic. See code at https://github.com/msed-Ebrahimi/GIF

Figures (7)

• Figure 1: a) Illustrating the growth in the number of identities in the FR datasets over time. b) Conventional scalar label leads to the linear association between computational cost and the number of identities. $\mathcal{D} = \{ (\mathbf{x}_i, y_i) \in \mathcal{X}\times \mathcal{Y}\}$ is the training benchmark, $m$ number of identities, and $\mathcal{V}$ is set of all possible codes. c) Comparing the increase in the computational cost of Fully Connected (FC) deng2019arcface, PFC an2022killing, DCQ li2021dynamic, F$^2$C wang2022efficient and GIF as the number of identities increases. GIF significantly reduces the growth rate by changing the scaling from linear to logarithmic.
• Figure 2: a) We first convert scalar labels to identity codes. We use CLIP radford2021learning visual encoder to initialize code vectors. Circles with the same colors represent identities with similar generic information. b) Our framework utilizes the identity codes to train the FR model. $\mathcal{D} = \{ (\mathbf{x}_i, y_i) \in \mathcal{X}\times \mathcal{Y}\}$ is the training benchmark, and $\mathcal{V}$ denotes set of all possible codes.
• Figure 3: Left) Overview of proposed tokenization. We position each $y_i$ on the $\mathcal{S}^{d-1}$ using hyperspherical code vectors $\mathbf{h}_{y_i}$ in a way that the pair-wise distance among arbitrary $\mathbf{h}_{y_i}$ and $\mathbf{h}_{y_j}$ is maximized. Then, we construct identity codes by applying hierarchical clustering over $\mathbf{H}$. Right) Overview of proposed FR training pipeline.
• Figure 4: The maximum ${\uparrow}$, minimum ${\uparrow}$, and mean ${\uparrow}$ pairwise cosine distance among Softmax centroids of Fully Connected (FC) ArcFace (FC) deng2019arcface, PFC an2022killing, and the code vectors $\mathbf{h}_i$ of our proposal when $d=512$. More separation among $\mathbf{h}_i$ reflects better metric-space exploitation, leading to more discrimination power in the embedding of $F_{\theta}$. The separation of Softmax centroids is based on the final training checkpoint.
• Figure 5: (a, b) GPU memory consumption and Training speed comparison between FC, PFC and GIF: GIF significantly improves training cost compared to both FC deng2019arcface and PFC an2022killing. (c) Ablation on loss components when the training data is MS1MV2, backbone is ResNet-100. (d) Comparing the evaluation performance between employing structured vs. atomic identity codes during training: training using atomic code fails. (e) Effect of the length of codes in the performance when the dataset and label space changes. The average performance across LFW, CFP-FP, CPLFW, CALFW, and AgeDB is reported. (f) The range of tokens, i.e., $v$, when the $l$ changes across datasets.