Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Feature Alignment: Discovering Biased and Interpretable Subpopulations in Face Recognition Models

Ignacio Serna

TL;DR

Face recognition systems exhibit systematic biases across subpopulations, and relying on predefined attribute labels for auditing is costly and incomplete. Latent Feature Alignment (LFA) identifies semantically coherent subpopulations by aligning embeddings along latent directions in the embedding space, using a greedy expansion with a stopping threshold. LFA delivers higher semantic coherence than standard clustering, uncovers interpretable directions aligned with demographic attributes, and enables bias discovery without attribute annotations, validated across four backbones (ArcFace, CosFace, ElasticFace, PartialFC) and two benchmarks (RFW, CelebA) via impostor/genuine similarity analyses and FMR-based metrics. This label-free auditing tool enhances practical fairness assessment and can be extended to other embedding-based domains beyond face recognition.

Abstract

Modern face recognition models achieve high overall accuracy but continue to exhibit systematic biases that disproportionately affect certain subpopulations. Conventional bias evaluation frameworks rely on labeled attributes to form subpopulations, which are expensive to obtain and limited to predefined categories. We introduce Latent Feature Alignment (LFA), an attribute-label-free algorithm that uses latent directions to identify subpopulations. This yields two main benefits over standard clustering: (i) semantically coherent grouping, where faces sharing common attributes are grouped together more reliably than by proximity-based methods, and (ii) discovery of interpretable directions, which correspond to semantic attributes such as age, ethnicity, or attire. Across four state-of-the-art recognition models (ArcFace, CosFace, ElasticFace, PartialFC) and two benchmarks (RFW, CelebA), LFA consistently outperforms k-means and nearest-neighbor search in intra-group semantic coherence, while uncovering interpretable latent directions aligned with demographic and contextual attributes. These results position LFA as a practical method for representation auditing of face recognition models, enabling practitioners to identify and interpret biased subpopulations without predefined attribute annotations.

Latent Feature Alignment: Discovering Biased and Interpretable Subpopulations in Face Recognition Models

TL;DR

Face recognition systems exhibit systematic biases across subpopulations, and relying on predefined attribute labels for auditing is costly and incomplete. Latent Feature Alignment (LFA) identifies semantically coherent subpopulations by aligning embeddings along latent directions in the embedding space, using a greedy expansion with a stopping threshold. LFA delivers higher semantic coherence than standard clustering, uncovers interpretable directions aligned with demographic attributes, and enables bias discovery without attribute annotations, validated across four backbones (ArcFace, CosFace, ElasticFace, PartialFC) and two benchmarks (RFW, CelebA) via impostor/genuine similarity analyses and FMR-based metrics. This label-free auditing tool enhances practical fairness assessment and can be extended to other embedding-based domains beyond face recognition.

Abstract

Modern face recognition models achieve high overall accuracy but continue to exhibit systematic biases that disproportionately affect certain subpopulations. Conventional bias evaluation frameworks rely on labeled attributes to form subpopulations, which are expensive to obtain and limited to predefined categories. We introduce Latent Feature Alignment (LFA), an attribute-label-free algorithm that uses latent directions to identify subpopulations. This yields two main benefits over standard clustering: (i) semantically coherent grouping, where faces sharing common attributes are grouped together more reliably than by proximity-based methods, and (ii) discovery of interpretable directions, which correspond to semantic attributes such as age, ethnicity, or attire. Across four state-of-the-art recognition models (ArcFace, CosFace, ElasticFace, PartialFC) and two benchmarks (RFW, CelebA), LFA consistently outperforms k-means and nearest-neighbor search in intra-group semantic coherence, while uncovering interpretable latent directions aligned with demographic and contextual attributes. These results position LFA as a practical method for representation auditing of face recognition models, enabling practitioners to identify and interpret biased subpopulations without predefined attribute annotations.
Paper Structure (20 sections, 4 equations, 13 figures, 7 tables, 2 algorithms)

This paper contains 20 sections, 4 equations, 13 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Supervised Approaches
  4. Unsupervised Approaches
  5. Methodology
  6. Latent Feature Alignment (LFA)
  7. Automated Discovery of Interpretable and Biased Groups
  8. Experimental Evaluation
  9. Datasets and Models
  10. Semantic Coherence of Groups
  11. Interpretability of Latent Directions
  12. Bias Discovery with Attribute Annotations
  13. Conclusion and Discussion
  14. Limitations
  15. RFW Annotation and Validation
  16. ...and 5 more sections

Figures (13)

  • Figure 1: Left: Removal of semantic attribtues from face images. For each group (A, B, C and D) the top image shows the inversion (embedding decoded) of the original image and the bottom one the original image after traversing back the embedding along the latent direction of the group. Right: Addition of semantic attributes. First column shows the original image, second column the inversion, and the rest of the columns (A-D) the original image decoded after traversing it along the latent direction of groups A-D.
  • Figure 2: Attribute probability vs traversal strength. Predicted probabilities from an attribute classifier for images traversed along the latent direction of Group A (young, white, blonde, female). Relevant attributes (e.g., Blond Hair, Young, Female, White) change monotonically with traversal strength, while unrelated attributes (e.g., Wearing Hat, Fully Visible Forehead) remain stable.
  • Figure 3: Qualitative and quantitative evidence of bias discovered by LFA on ArcFace.Left: Example images from different groups (R: random sample, A: Caucasian blonde females, B: young Indian women, C: Indian males with turbans, D: Chinese men with mustaches in black-and-white photographs). Center: Similarity score distributions for groups R–D. The right distribution of each group corresponds to genuine pairs (higher similarity), while the left one represents impostor pairs (lower similarity). Darker colors indicate higher error rates. Right: False Match Rate (FMR) curves for groups R–D, computed from genuine and impostor distributions (center). The extent of bias is indicated by the hue of the colour used; darker hues denote a higher degree of bias.
  • Figure 4: Distribution of cosine similarity scores for impostor and genuine comparisons within the same group.
  • Figure 5: False Match Rate (FMR) curves for each face recognition model for all groups.
  • ...and 8 more figures