Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurement

TL;DR

This work addresses how train/test identity overlap can produce optimistic accuracy in face recognition benchmarks. By reverse-engineering LFW–MS1MV2 overlap and establishing a uniform training framework, the authors quantify the extent of identity overlap (46.93% of LFW identities) and demonstrate that overlap yields consistent accuracy advantages, especially on harder test sets. They further show that cleaning identity labels reduces noise and that identity-disjoint training is essential for fair benchmarking. The findings have practical implications for how face recognition models are trained and evaluated, suggesting that current benchmarks may overestimate true generalization if overlap is not controlled. Overall, the paper advocates for strict identity-disjoint evaluation to enable accurate cross-method comparisons and robust performance assessments.

Abstract

A fundamental tenet of pattern recognition is that overlap between training and testing sets causes an optimistic accuracy estimate. Deep CNNs for face recognition are trained for N-way classification of the identities in the training set. Accuracy is commonly estimated as average 10-fold classification accuracy on image pairs from test sets such as LFW, CALFW, CPLFW, CFP-FP and AgeDB-30. Because train and test sets have been independently assembled, images and identities in any given test set may also be present in any given training set. In particular, our experiments reveal a surprising degree of identity and image overlap between the LFW family of test sets and the MS1MV2 training set. Our experiments also reveal identity label noise in MS1MV2. We compare accuracy achieved with same-size MS1MV2 subsets that are identity-disjoint and not identity-disjoint with LFW, to reveal the size of the optimistic bias. Using more challenging test sets from the LFW family, we find that the size of the optimistic bias is larger for more challenging test sets. Our results highlight the lack of and the need for identity-disjoint train and test methodology in face recognition research.

Figures (6)

• Figure 1: Importance of identity-disjoint training in face recognition. It involves five test sets, which have same identities, for different face recognition tasks. On the vertical axis, "importance" of identity-disjoint training is the accuracy difference between identity-overlapped and identity-disjoint training. On the horizontal axis, "difficulty" of the test set is the average accuracy value of two training approaches.
• Figure 2: Examples of the same identities and the same images occur in both LFW and MS1MV2. More examples are shown in Figures 4-7 of Supplementary Material.
• Figure 3: Examples of the high similarity but marked as not the same identity (HSNS) and the low similarity but marked as the same identity (LSTS).
• Figure 4: Similarity distribution of top two matches in MS1MV2. Based on the manual marking result, the similarity distribution of the pairs marked as same identity and different identities are at middle and right.
• Figure 5: Examples of identity mis-categorization in MS1MV2. Top row: images from MS1MV2 identity folder 0_5873183; middle row: images from MS1MV2 folder 0_5876541; bottom row: images from MS1MV2 folder 0_5898354. The three "identities" are all Arnold Schwarzenagger.