Perceptual Fairness in Image Restoration

TL;DR

This work offers an alternative approach towards fairness in image restoration, by considering the Group Perceptual Index (GPI), which is defined as the statistical distance between the distribution of the group's ground truth images and the distribution of their reconstructions.

Abstract

Fairness in image restoration tasks is the desire to treat different sub-groups of images equally well. Existing definitions of fairness in image restoration are highly restrictive. They consider a reconstruction to be a correct outcome for a group (e.g., women) only if it falls within the group's set of ground truth images (e.g., natural images of women); otherwise, it is considered entirely incorrect. Consequently, such definitions are prone to controversy, as errors in image restoration can manifest in various ways. In this work we offer an alternative approach towards fairness in image restoration, by considering the Group Perceptual Index (GPI), which we define as the statistical distance between the distribution of the group's ground truth images and the distribution of their reconstructions. We assess the fairness of an algorithm by comparing the GPI of different groups, and say that it achieves perfect Perceptual Fairness (PF) if the GPIs of all groups are identical. We motivate and theoretically study our new notion of fairness, draw its connection to previous ones, and demonstrate its utility on state-of-the-art face image restoration algorithms.

Figures (32)

• Figure 1: Illustrative example of the proposed notion of Perceptual Fairness (PF). This figure presents four possible restoration algorithms exhibiting different behaviors and fairness performance. In this example, the sensitive attribute $A$ takes the values $0$ or $1$ with probabilities ${P(A=0)<P(A=1)}$. The distributions $p_{X}$ and $p_{Y}$ correspond to the ground truth signals (e.g., natural images) and their degraded measurements (e.g., noisy images), respectively. The distribution ${p_{X|A}(\cdot|a)}$ corresponds to the ground truth signals associated with the attribute value $a$, and ${p_{Y|A}(\cdot|a)}$ is the distribution of their degraded measurements. The distribution of all reconstructions is denoted by ${p_{\hat{X}}}$, and ${p_{\hat{X}|A}(\cdot|a)}$ is the distribution of the reconstructions associated with attribute value $a$. The Group Perceptual Index (GPI) of the group associated with $a$ is defined as the statistical distance between ${p_{\hat{X}|A}(\cdot|a)}$ and ${p_{X|A}(\cdot|a)}$, and good PF is achieved when the GPIs of all groups are (roughly) similar. For example, $\hat{X}_{1}$ achieves good PF since the GPIs of both $a=0$ and $a=1$ are roughly equal, while $\hat{X}_{3}$ achieves poor PF since the GPI of $a=0$ is worse (larger) than that of $a=1$. See \ref{['section:problem-formulation']} for more details.
• Figure 2: Examining fairness in face image super-resolution techniques through the lens of RDP pmlr-v139-jalal21b or PF (our proposed notion of fairness). Both RDP and PF assess how well an algorithm treats different fairness groups. Specifically, RDP evaluates the parity in the GP of different groups (higher GP is better), and PF evaluates the parity in the GPI of different groups (lower GPI is better). The results show that the groups old&Asian and old&non-Asian attain similar treatment according to RDP (similar GP scores that are roughly zero), while the latter group attains better treatment according to PF. In \ref{['section:experiments', 'appendix:disentangle_age_and_ethnicity']}, we show why this outcome of PF is the desired one.
• Figure 3: Comparison of the GP and the ${\text{GPI}_{\text{KID}}}$ of different fairness groups, using various state-of-the-art face image super-resolution methods. In most experiments, ${\text{GPI}_{\text{KID}}}$ suggests a fairness discrepancy between the groups old&non-Asian and old&Asian, while the GP of these groups is roughly equal.
• Figure 4: Using adversarial attacks to inject bias into the outputs of RestoreFormer++, in a setting where it (roughly) satisfies RDP. Such attacks are detected by PF but not by RDP.
• Figure 5: Illustration of \ref{['example:toy-dmax']}. Left: Conditional probability density functions $p_{X|A}(\cdot|a),\,p_{\hat{X}_{\text{MSE}}|A}(\cdot|a),\,p_{\hat{X}_{\text{Posterior}}|A}(\cdot|a),$ and $p_{\hat{X}_{\text{MSE+PI}}|A}(\cdot|a)$, where $a=1$ (left plot) or $a=0$ (right plot). Right: The ${\text{GPI}_{d_{\text{TV}}}}$ and ${\text{GPI}_{W_{1}}}$ of each group (associated with $a=1$ or $a=0$). The dotted lines ${\text{PF}_{d_{\text{TV}}}}$ or ${\text{PF}_{W_{1}}}$ correspond to the points where perfect ${\text{PF}_{d_{\text{TV}}}}$ or perfect ${\text{PF}_{W_{1}}}$ is achieved, respectively. It is clear that all three estimators achieve sub-optimal ${\text{PF}_{d_{\text{TV}}}}$ and sub-optimal ${\text{PF}_{W_{1}}}$. See \ref{['appendix:toy']} for more details.

Theorems & Definitions (13)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Example 1
• Theorem 4
• proof
• Theorem 4
• proof
• Theorem 4