Efficient Expression Neutrality Estimation with Application to Face Recognition Utility Prediction

TL;DR

The paper addresses quantifying facial expression neutrality as a component of face image quality to improve interoperability of biometric FR systems. It proposes an efficient estimator that uses intermediate features from lightweight expression recognition models with $5.3$M and $9.3$M parameters (HSE-1 and HSE-2) to train two-class neutral vs non-neutral classifiers. The study compares three classifiers (SVM, Random Forest, AdaBoost) across eight datasets, evaluating both expression classification performance and FR utility prediction using DET and EDC curves, and finds that RF and AdaBoost excel at neutral detection while SVM yields the best utility prediction (e.g., pAUC around $1.64\%$), highlighting that strong neutral classification does not always translate to better utility estimation. These results inform practical deployment and calibration under ISO/CD3 29794-5, guiding operators to tailor neutrality detectors to application needs and dataset biases.

Abstract

The recognition performance of biometric systems strongly depends on the quality of the compared biometric samples. Motivated by the goal of establishing a common understanding of face image quality and enabling system interoperability, the committee draft of ISO/IEC 29794-5 introduces expression neutrality as one of many component quality elements affecting recognition performance. In this study, we train classifiers to assess facial expression neutrality using seven datasets. We conduct extensive performance benchmarking to evaluate their classification and face recognition utility prediction abilities. Our experiments reveal significant differences in how each classifier distinguishes "neutral" from "non-neutral" expressions. While Random Forests and AdaBoost classifiers are most suitable for distinguishing neutral from non-neutral facial expressions with high accuracy, they underperform compared to Support Vector Machines in predicting face recognition utility.

Figures (6)

• Figure 1: Overview of our two-class classification approach utilizing features extracted from a pre-trained expression recognition model (HSE Savchenko-HSE-SISY-2021). We redefine the confidence scores of the neutral class as an efficient measure for facial expression neutrality.
• Figure 2: Illustration of our feature extraction based on the expression recognition models of Savanchenko Savchenko-HSE-SISY-2021: HSE-1 (EfficientNet-b0 Tan-EfficientNet-PMLR-2021 and HSE-2 (EfficientNet-b2 Tan-EfficientNet-PMLR-2021). The discrete bars represent the final Softmax scores, while the continuous bars denote the intermediate feature embeddings $\mathcal{F}_{\text{HSE-1}}\in \mathbb{R}^{1280}$, $\mathcal{F}_{\text{HSE-2}}\in \mathbb{R}^{1408}$. Figure \ref{['fig:feature-combination']} demonstrates how the features are combined to train our classifiers.
• Figure 3: Overview of feature combinations employed in training our two-class classifiers to distinguish neutral from non-neutral facial expressions. The illustrated features correspond to the feature extraction and color scheme depicted in Figure \ref{['fig:feature-extraction']}.
• Figure 4: DET curves illustrating the classification performance benchmarking of the classifiers across six feature combinations conducted on the evaluation dataset.
• Figure 5: EDC curves depicting the FR utility prediction benchmarking of the classifiers across six feature combinations conducted on the evaluation dataset.