Feature Transfer Learning for Deep Face Recognition with Under-Represented Data

TL;DR

Face recognition suffers when many subjects are under-represented (UR) with few samples. We propose Feature Transfer Learning (FTL), which transfers intra-class variance from regular classes to UR classes under a Gaussian variance prior across subjects, implemented via a center-based transfer module and aided by an alternating two-stage training regime and a metric $L_2$ regularization. The approach yields strong gains on LFW, IJB-A, and MS-Celeb-1M, supports one-shot/low-shot scenarios, and enables smooth feature interpolation that disentangles identity from non-identity variations. Together, these contributions demonstrate effective use of UR data to reduce classifier bias and improve discrimination, with potential applicability to other fine-grained recognition tasks.

Abstract

Despite the large volume of face recognition datasets, there is a significant portion of subjects, of which the samples are insufficient and thus under-represented. Ignoring such significant portion results in insufficient training data. Training with under-represented data leads to biased classifiers in conventionally-trained deep networks. In this paper, we propose a center-based feature transfer framework to augment the feature space of under-represented subjects from the regular subjects that have sufficiently diverse samples. A Gaussian prior of the variance is assumed across all subjects and the variance from regular ones are transferred to the under-represented ones. This encourages the under-represented distribution to be closer to the regular distribution. Further, an alternating training regimen is proposed to simultaneously achieve less biased classifiers and a more discriminative feature representation. We conduct ablative study to mimic the under-represented datasets by varying the portion of under-represented classes on the MS-Celeb-1M dataset. Advantageous results on LFW, IJB-A and MS-Celeb-1M demonstrate the effectiveness of our feature transfer and training strategy, compared to both general baselines and state-of-the-art methods. Moreover, our feature transfer successfully presents smooth visual interpolation, which conducts disentanglement to preserve identity of a class while augmenting its feature space with non-identity variations such as pose and lighting.

Figures (8)

• Figure 1: Illustration of the UR data problem and our proposed solution. (a) The data distribution of CASIA-WebFace dataset yi2014learning. (b) Classifier weight norm varies across classes in proportion to their volume. (c) Weight norm for regular class $1008$ is larger than UR class $10449$, causing a bias in the decision boundary (dashed line) towards ID $10449$. (d) Data re-sampling solves the classifier bias to some extent. However, the variance of ID 1008 is much larger than ID $10449$. We augment the feature space of ID $1008$ (dashed ellipsoid) and propose improved training strategies, which corrects the classifier bias and learns a better feature representation.
• Figure 2: Overview of our proposed FTL framework. It consists of a feature extractor $Enc$, a decoder $Dec$, a feature filter $R$, a fully connected layer as classifier $FC$, and a feature transfer module $G$. The network is trained with an alternating bi-stage strategy. At stage $1$ (solid arrows), we fix $Enc$ and apply feature transfer $G$ to generate new feature samples (blue triangles) that are more diverse to reshape the decision boundary. In stage $2$ (dashed arrows), we fix the rectified classifier $FC$, and update all the other models. As a result, the samples that are originally on or across the boundary are pushed towards their center (blue arrows in bottom right). Best viewed in color.
• Figure 3: Visualization of samples closest to the feature center of classes with most number of images (left) and classes with least number of images (right). We find that near-frontal close-to-neutral faces are the nearest neighbors of the feature centers of regular classes. However, the nearest neighbors of the feature centers of UR classes still contain pose and expression variations. Features are extracted by VGGFace model ParkhiVGG and samples are from CASIA-WebFace dataset.
• Figure 4: (a) Center estimation error comparison. (b) Illustration of intra- and inter-class variances. Circles from small to large show the minimum, mean and maximum distances from intra-class samples to center. Distances are averaged across $1$K classes.
• Figure 5: Toy example on MNIST to show the effectiveness of our m-$L_2$ regularization. Figure shows the feature distributions for models trained without (a) and with (b) m-$L_2$ regularization.