Impact of Sunglasses on One-to-Many Facial Identification Accuracy

TL;DR

This paper addresses the accuracy degradation of one-to-many facial identification when the probe image contains sunglasses, a common occlusion in surveillance. It uses a novel ND-sunglasses dataset and controlled degradations to quantify how sunglasses, alone and with blur or low resolution, shift mated/non-mated score distributions for AdaFace and ArcFace. It demonstrates practical mitigations: adding sunglasses to gallery images partially recovers lost accuracy without re-training, and increasing the representation of wearing-sunglasses images in training further reduces FPIR; synthetic sunglasses via Vec2Face also shows promise. The work provides a dataset for replication and yields objective criteria for evaluating probe quality in real-world deployment, with implications for reducing wrongful arrests.

Abstract

One-to-many facial identification is documented to achieve high accuracy in the case where both the probe and the gallery are "mugshot quality" images. However, an increasing number of documented instances of wrongful arrest following one-to-many facial identification have raised questions about its accuracy. Probe images used in one-to-many facial identification are often cropped from frames of surveillance video and deviate from "mugshot quality" in various ways. This paper systematically explores how the accuracy of one-to-many facial identification is degraded by the person in the probe image choosing to wear dark sunglasses. We show that sunglasses degrade accuracy for mugshot-quality images by an amount similar to strong blur or noticeably lower resolution. Further, we demonstrate that the combination of sunglasses with blur or lower resolution results in even more pronounced loss in accuracy. These results have important implications for developing objective criteria to qualify a probe image for the level of accuracy to be expected if it used for one-to-many identification. To ameliorate the accuracy degradation caused by dark sunglasses, we show that it is possible to recover about 38% of the lost accuracy by synthetically adding sunglasses to all the gallery images, without model re-training. We also show that the frequency of wearing-sunglasses images is very low in existing training sets, and that increasing the representation of wearing-sunglasses images can greatly reduce the error rate. The image set assembled for this research is available at https://cvrl.nd.edu/projects/data/ to support replication and further research.

Figures (20)

• Figure 1: How much does wearing sunglasses degrade one-to-many facial identification accuracy? One-to-many identification has high accuracy when the probe and gallery images are all high quality Grother_NIST_2019. This work explores how accuracy degrades when the person in the probe image wears sunglasses, and alternatives for ameliorating this problem.
• Figure 2: The pipeline for creating sunglasses-added versions of original images. An Android emulator and ManyCam are used to handle the image input and processing steps, the FaceLab app adds sunglasses, and a screenshot of the FaceLab results is stored.
• Figure 3: Examples of original images and synthetic versions with sunglasses added using FaceLab app. Adding synthetic sunglasses alters only the sunglasses region of the original image. Note that the style of the synthetic sunglasses varies between images.
• Figure 4: Examples of FaceLab's synthetic sunglasses on AR dataset. This 2x4 matrix displays two subjects (rows) under four conditions (columns): normal expression, smiling expression, real sunglasses, and synthetic sunglasses applied to the normal expression.
• Figure 5: Cosine similarity distributions comparing images with smiling expressions to those with real and synthetic sunglasses. The observed shift towards higher similarity scores with synthetic sunglasses, compared to real ones, shows that the FaceLab app's sunglasses addition can preserve identity.