Distributionally Generative Augmentation for Fair Facial Attribute Classification

TL;DR

This work proposes a novel, generation-based two-stage framework to train a fair FAC model on biased data without additional annotation, and enhances interpretability by explicitly showing the spurious attributes in image space.

Abstract

Facial Attribute Classification (FAC) holds substantial promise in widespread applications. However, FAC models trained by traditional methodologies can be unfair by exhibiting accuracy inconsistencies across varied data subpopulations. This unfairness is largely attributed to bias in data, where some spurious attributes (e.g., Male) statistically correlate with the target attribute (e.g., Smiling). Most of existing fairness-aware methods rely on the labels of spurious attributes, which may be unavailable in practice. This work proposes a novel, generation-based two-stage framework to train a fair FAC model on biased data without additional annotation. Initially, we identify the potential spurious attributes based on generative models. Notably, it enhances interpretability by explicitly showing the spurious attributes in image space. Following this, for each image, we first edit the spurious attributes with a random degree sampled from a uniform distribution, while keeping target attribute unchanged. Then we train a fair FAC model by fostering model invariance to these augmentation. Extensive experiments on three common datasets demonstrate the effectiveness of our method in promoting fairness in FAC without compromising accuracy. Codes are in https://github.com/heqianpei/DiGA.

Figures (10)

• Figure 1: FAC models can be unfair by exhibiting accuracy inconsistencies across varied data subpopulations (e.g., 95.4% accuracy on Smiling&Female and 70.1% accuracy on Smiling&Male). This unfairness is predominantly attributed to data bias, measured by $\beta$. In general, the more biased the data (i.e., larger $\beta$), the more unfair the model. Most of existing methods such as IRM arjovsky2019invariant and resampling romano2020achieving rely on the labels of spurious attributes. Our method can improve the fairness, measured by EO and the worst-group accuracy (Eq. (\ref{['EO']}) and (\ref{['worst']})), of FAC models without additional annotations. Experiments above are performed on CelebA liu2018large.
• Figure 2: Moving the latent codes$\boldsymbol{z}$ of a well-trained generative model in a learned direction$\boldsymbol{n}$ can edit the target attribute (e.g., Smiling) of images ($\boldsymbol{x}\rightarrow \boldsymbol{x}_y$) shen2020interfacegan. We observe that if the data is biased, the learned direction will be biased by containing information of spurious attributes (e.g., Male) ($\boldsymbol{x}\rightarrow \boldsymbol{x}_1,\boldsymbol{x}_2$). Based on this, we synthesize a direction $\boldsymbol{n}_s$ to manipulate the spurious attributes while keeping target attribute unchanged ($\boldsymbol{x}\rightarrow \boldsymbol{x}_s$).
• Figure 3: Illustration of our bias detection method. (a) Latent codes for well-trained generative models encode disentangled representations, and moving the latent codes along the normal vector $\boldsymbol{n}$ of the learned classification boundary $\boldsymbol{w}^*$ can edit the target attribute of images. (b) The learned boundary $\boldsymbol{w}^*$ will be biased if the training data is biased, and the bias degree of boundary $\boldsymbol{w}^*$ is influenced by the regularization strength $\lambda$. (c) By choosing the appropriate coefficients $(c^*_1,c^*_2)$, we can combine two biased boundaries $(\boldsymbol{w}^*_1,\boldsymbol{w}^*_2)$ into a new boundary $\boldsymbol{w}_{cmb}$ that is only dependent of the spurious attributes. So the direction $\boldsymbol{n}_{cmb}$, the normal vector of $\boldsymbol{w}_{cmb}$, only encode the semantics of spurious attributes. (d) To find the optimal coefficients $(c^*_1,c^*_2)$, we perform grid search with the help of a reference model.
• Figure 4: Distributionally generative augmentation for fair representation learning. For each image, we edit its spurious attributes by using the combined semantic direction $\boldsymbol{n}_{cmb}$ in latent space. The editing degrees $\alpha'$ and $\alpha"$ are randomly sampled from a uniform distribution. We also perform traditional augmentation $T(\cdot)$ such as random clipping. The encoder is trained to learn fair and effective representations by closing the distance between augmented views. We use momentum encoder to avoid collapsing.
• Figure 5: Bias detection results on CelebA dataset. The constructed training dataset is biased, where 75% Smiling images are Female&Young and 75% Non-smiling images are Male&Non-young. We only have labels of target attribute Smiling. By utilizing the proposed bias detection method, we obtain the combined direction and edit training images. It can be observed that the changes of gender and age are faithfully reflected in image space, illustrating which attributes are spurious attributes explicitly and thus enhancing interpretability.

Theorems & Definitions (1)

• Theorem 1: Optimal combination coefficients' existence