Hairmony: Fairness-aware hairstyle classification

TL;DR

A novel hairstyle taxonomy developed in collaboration with a diverse group of domain experts is introduced which allows for explicit control of diversity of hairstyle attributes, hair colors, facial appearance, poses, environments and other parameters and is significantly more robust for challenging hairstyles than recent parametric approaches.

Abstract

We present a method for prediction of a person's hairstyle from a single image. Despite growing use cases in user digitization and enrollment for virtual experiences, available methods are limited, particularly in the range of hairstyles they can capture. Human hair is extremely diverse and lacks any universally accepted description or categorization, making this a challenging task. Most current methods rely on parametric models of hair at a strand level. These approaches, while very promising, are not yet able to represent short, frizzy, coily hair and gathered hairstyles. We instead choose a classification approach which can represent the diversity of hairstyles required for a truly robust and inclusive system. Previous classification approaches have been restricted by poorly labeled data that lacks diversity, imposing constraints on the usefulness of any resulting enrollment system. We use only synthetic data to train our models. This allows for explicit control of diversity of hairstyle attributes, hair colors, facial appearance, poses, environments and other parameters. It also produces noise-free ground-truth labels. We introduce a novel hairstyle taxonomy developed in collaboration with a diverse group of domain experts which we use to balance our training data, supervise our model, and directly measure fairness. We annotate our synthetic training data and a real evaluation dataset using this taxonomy and release both to enable comparison of future hairstyle prediction approaches. We employ an architecture based on a pre-trained feature extraction network in order to improve generalization of our method to real data and predict taxonomy attributes as an auxiliary task to improve accuracy. Results show our method to be significantly more robust for challenging hairstyles than recent parametric approaches.

Figures (6)

• Figure 1: Graphical overview of our proposed hair taxonomy consisting of global and regional attributes. Note that some values are not visualized for the 'Gathered' and 'Length' attributes.
• Figure 2: Example images from our synthetic training data showing a variety of hairstyles combined with random facial appearance, pose and environment.
• Figure 3: Training and operating scheme of the proposed model. Given an input image, frozen backbone $B$ produces intermediate features that are later used to produce intermediate representation via shared layer $FC_{L}$. Given the output of $FC_{L}$: (i) hairstyle prediction head $FC_{s}$ predicts the final hairstyle and (ii) hairstyle attribute prediction heads $FC_{1}, \ldots, FC_{A}$ predict taxonomic hairstyle attributes.
• Figure 4: Comparison of our method with recent parametric hair prediction approach HairStep Zheng_2023_CVPR. HairStep performs well for long straight hair (top row), but has a strong bias towards this hair style and type. This results in poor performance for short styles and coily or curly hair types, even if results appear to be of reasonable quality when viewed from the front. While our results provide less direct representation in some cases, they are significantly more robust across diverse hairstyles. HairStep results are manually aligned to reconstructed face meshes. FairFace images -- CC BY 4.0.
• Figure 5: Qualitative results for our method on the FairFace karkkainen2019fairface evaluation subset. Bottom row shows failure cases, specifically: missed hair, incorrect hair type/strand styling, incorrect gathering, incorrect length, hairstyle not in library. FairFace images -- CC BY 4.0.