Accurately Classifying Out-Of-Distribution Data in Facial Recognition

TL;DR

The goal was to make models not only more accurate but also more fair by scanning a more expanded range of images by incorporating the Outlier Exposure model and finding models utilizing outlier exposure could result in more fair classification.

Abstract

Standard classification theory assumes that the distribution of images in the test and training sets are identical. Unfortunately, real-life scenarios typically feature unseen data (``out-of-distribution data") which is different from data in the training distribution (``in-distribution"). This issue is most prevalent in social justice problems where data from under-represented groups may appear in the test data without representing an equal proportion of the training data. This may result in a model returning confidently wrong decisions and predictions. We are interested in the following question: Can the performance of a neural network improve on facial images of out-of-distribution data when it is trained simultaneously on multiple datasets of in-distribution data? We approach this problem by incorporating the Outlier Exposure model and investigate how the model's performance changes when other datasets of facial images were implemented. We observe that the accuracy and other metrics of the model can be increased by applying Outlier Exposure, incorporating a trainable weight parameter to increase the machine's emphasis on outlier images, and by re-weighting the importance of different class labels. We also experimented with whether sorting the images and determining outliers via image features would have more of an effect on the metrics than sorting by average pixel value, and found no conclusive results. Our goal was to make models not only more accurate but also more fair by scanning a more expanded range of images. Utilizing Python and the Pytorch package, we found models utilizing outlier exposure could result in more fair classification.

Figures (6)

• Figure 1: Any point inside the dashed circle gives $f(\mathbf{x}) = 1$ and its true label is class 1, while the true label of points in the rest of the whole space $[-6,6]^2$ is class 0. The training data $(\mathbf{x}^{\text{train}}, y)$ has inputs from the smaller square $\mathbf{x}^{\text{train}} \in [-1.5, 1.5]^2$, and $y = f(\mathbf{x}^{\text{train}})$. This can be seen in the middle small square with blue (class 1) in the overlapping part with $x_1^2 + x_2^2 \leq 4$; and red corners (class 0). The color bar on the side gives the predicted probability of the test data (a uniform mesh grid on $[-6,6]^2$) in class 1.
• Figure 1: How a CNN Processes an Image.
• Figure 1: ROC Curve from all experiments used, labelled in the order of "train set-test set-outlier set". Orange is the classifier performance and the dashed line is for 50% accuracy.
• Figure 2: A simple math example to illustrate outlier exposure. Three pairs of classification results in the toy example. From top to bottom: (a) is the result when training and testing data set is using the same normal distribution (in-distribution data). (b) is training on in-distribution data, and testing on out-of-distribution data. (c) Training on in-distribution data, and testing on out-of-distribution data with Outlier Exposure
• Figure 3: Distribution of Pixel Frequency in UTKFace $\&$ FairFace

Theorems & Definitions (2)

• Example 3.1
• Example 3.2