Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data quality dimensions for fair AI

Camilla Quaresmini, Giuseppe Primiero

TL;DR

The paper treats fairness in AI as a data-quality problem, arguing that mislabeling and evolving labels undermine fairness beyond pure accuracy. It introduces a temporal, data-quality–driven framework that extends completeness, consistency, and timeliness to bias mitigation, and formalizes time-aware metrics such as $Compl_{\,\mathcal{T}}(L(X))$, $Rel_{\,\mathcal{T}}(X)$, and $Fair_{\,\mathcal{T}}(X)$, with a change rate $\varepsilon_{\,\mathcal{T}}(X)$ bounded by a threshold $\pi$. By integrating concepts like the temporal confident joint $C_{\tilde{y},y^{*}}[\cdot,\mathcal{T}]$, the work demonstrates how label noise can evolve and affect fairness, and discusses implementing these ideas in tools such as Cleanlab and BRIO. The proposed approach advocates timeliness as a foundational dimension for building fairer AI classifications, especially in sensitive domains like gender classification, and outlines future work on empirical validation and extended probabilistic modeling of label dynamics. Overall, the paper offers a principled, time-aware lens to assess and improve data quality and fairness in AI systems.

Abstract

Artificial Intelligence (AI) systems are not intrinsically neutral and biases trickle in any type of technological tool. In particular when dealing with people, the impact of AI algorithms' technical errors originating with mislabeled data is undeniable. As they feed wrong and discriminatory classifications, these systems are not systematically guarded against bias. In this article we consider the problem of bias in AI systems from the point of view of data quality dimensions. We highlight the limited model construction of bias mitigation tools based on accuracy strategy, illustrating potential improvements of a specific tool in gender classification errors occurring in two typically difficult contexts: the classification of non-binary individuals, for which the label set becomes incomplete with respect to the dataset; and the classification of transgender individuals, for which the dataset becomes inconsistent with respect to the label set. Using formal methods for reasoning about the behavior of the classification system in presence of a changing world, we propose to reconsider the fairness of the classification task in terms of completeness, consistency, timeliness and reliability, and offer some theoretical results.

Data quality dimensions for fair AI

TL;DR

The paper treats fairness in AI as a data-quality problem, arguing that mislabeling and evolving labels undermine fairness beyond pure accuracy. It introduces a temporal, data-quality–driven framework that extends completeness, consistency, and timeliness to bias mitigation, and formalizes time-aware metrics such as , , and , with a change rate bounded by a threshold . By integrating concepts like the temporal confident joint , the work demonstrates how label noise can evolve and affect fairness, and discusses implementing these ideas in tools such as Cleanlab and BRIO. The proposed approach advocates timeliness as a foundational dimension for building fairer AI classifications, especially in sensitive domains like gender classification, and outlines future work on empirical validation and extended probabilistic modeling of label dynamics. Overall, the paper offers a principled, time-aware lens to assess and improve data quality and fairness in AI systems.

Abstract

Artificial Intelligence (AI) systems are not intrinsically neutral and biases trickle in any type of technological tool. In particular when dealing with people, the impact of AI algorithms' technical errors originating with mislabeled data is undeniable. As they feed wrong and discriminatory classifications, these systems are not systematically guarded against bias. In this article we consider the problem of bias in AI systems from the point of view of data quality dimensions. We highlight the limited model construction of bias mitigation tools based on accuracy strategy, illustrating potential improvements of a specific tool in gender classification errors occurring in two typically difficult contexts: the classification of non-binary individuals, for which the label set becomes incomplete with respect to the dataset; and the classification of transgender individuals, for which the dataset becomes inconsistent with respect to the label set. Using formal methods for reasoning about the behavior of the classification system in presence of a changing world, we propose to reconsider the fairness of the classification task in terms of completeness, consistency, timeliness and reliability, and offer some theoretical results.
Paper Structure (9 sections, 2 theorems, 7 equations, 2 figures, 2 tables)

This paper contains 9 sections, 2 theorems, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Fairness and Bias in ML
  3. Mitigating Bias
  4. Extending Data Dimensions for Fair AI
  5. Incomplete Label Set and Inconsistent Labeling
  6. Enter Time
  7. Back to Cleanlab
  8. Temporal-based Fairness in AI
  9. Conclusion

Key Result

Theorem 1

Given a label set $L$ complete at time $t$, a classification algorithm guarantees a fair classification at time $t'>t$ if and only if the change rate determined with respect to $L$ is $\epsilon < \pi$.

Figures (2)

  • Figure 1: Confusion matrix at time $n$.
  • Figure 2: Confusion matrix at time $n+m$.

Theorems & Definitions (7)

  • Definition 1: Completeness of a label set
  • Definition 2: Reliability of a classification algorithm
  • Definition 3: Fairness for AI classification systems
  • Theorem 1
  • proof
  • Theorem 2
  • proof