Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road of Building Responsible AI Systems

Gordon Dai, Yunze Xiao

TL;DR

The paper reframes Responsible AI evaluation by treating metric inconsistency as a feature rather than a flaw, arguing that intra-concept inconsistencies (e.g., among different fairness metrics) and inter-concept trade-offs (e.g., accuracy versus privacy) preserve diverse stakeholder values, enrich concept understanding, and act as practical regularizers. It formalizes two forms of inconsistency, reviews canonical results and mechanisms (gradient conflicts, Pareto/Rashomon sets, and cross-metric regularization) that can yield robust, generalizable behavior under real-world uncertainty, and introduces a protocol for transparent documentation of inconsistency via Metric Provenance Sheets and SPHERE-aligned reporting. The authors advocate practice-driven theories, empirical study of human–metric interactions, and profile-based governance to navigate pluralistic evaluation without sacrificing accountability. Overall, the work argues for embracing contradiction to enhance normative representation, epistemic fidelity, and operational resilience in responsible AI systems.

Abstract

This position paper argues that the theoretical inconsistency often observed among Responsible AI (RAI) metrics, such as differing fairness definitions or tradeoffs between accuracy and privacy, should be embraced as a valuable feature rather than a flaw to be eliminated. We contend that navigating these inconsistencies, by treating metrics as divergent objectives, yields three key benefits: (1) Normative Pluralism: Maintaining a full suite of potentially contradictory metrics ensures that the diverse moral stances and stakeholder values inherent in RAI are adequately represented. (2) Epistemological Completeness: The use of multiple, sometimes conflicting, metrics allows for a more comprehensive capture of multifaceted ethical concepts, thereby preserving greater informational fidelity about these concepts than any single, simplified definition. (3) Implicit Regularization: Jointly optimizing for theoretically conflicting objectives discourages overfitting to one specific metric, steering models towards solutions with enhanced generalization and robustness under real-world complexities. In contrast, efforts to enforce theoretical consistency by simplifying or pruning metrics risk narrowing this value diversity, losing conceptual depth, and degrading model performance. We therefore advocate for a shift in RAI theory and practice: from getting trapped in inconsistency to characterizing acceptable inconsistency thresholds and elucidating the mechanisms that permit robust, approximated consistency in practice.

Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road of Building Responsible AI Systems

TL;DR

The paper reframes Responsible AI evaluation by treating metric inconsistency as a feature rather than a flaw, arguing that intra-concept inconsistencies (e.g., among different fairness metrics) and inter-concept trade-offs (e.g., accuracy versus privacy) preserve diverse stakeholder values, enrich concept understanding, and act as practical regularizers. It formalizes two forms of inconsistency, reviews canonical results and mechanisms (gradient conflicts, Pareto/Rashomon sets, and cross-metric regularization) that can yield robust, generalizable behavior under real-world uncertainty, and introduces a protocol for transparent documentation of inconsistency via Metric Provenance Sheets and SPHERE-aligned reporting. The authors advocate practice-driven theories, empirical study of human–metric interactions, and profile-based governance to navigate pluralistic evaluation without sacrificing accountability. Overall, the work argues for embracing contradiction to enhance normative representation, epistemic fidelity, and operational resilience in responsible AI systems.

Abstract

This position paper argues that the theoretical inconsistency often observed among Responsible AI (RAI) metrics, such as differing fairness definitions or tradeoffs between accuracy and privacy, should be embraced as a valuable feature rather than a flaw to be eliminated. We contend that navigating these inconsistencies, by treating metrics as divergent objectives, yields three key benefits: (1) Normative Pluralism: Maintaining a full suite of potentially contradictory metrics ensures that the diverse moral stances and stakeholder values inherent in RAI are adequately represented. (2) Epistemological Completeness: The use of multiple, sometimes conflicting, metrics allows for a more comprehensive capture of multifaceted ethical concepts, thereby preserving greater informational fidelity about these concepts than any single, simplified definition. (3) Implicit Regularization: Jointly optimizing for theoretically conflicting objectives discourages overfitting to one specific metric, steering models towards solutions with enhanced generalization and robustness under real-world complexities. In contrast, efforts to enforce theoretical consistency by simplifying or pruning metrics risk narrowing this value diversity, losing conceptual depth, and degrading model performance. We therefore advocate for a shift in RAI theory and practice: from getting trapped in inconsistency to characterizing acceptable inconsistency thresholds and elucidating the mechanisms that permit robust, approximated consistency in practice.

Paper Structure

This paper contains 28 sections, 8 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Conceptual Framework: Two Forms of Metric Inconsistency
  3. Fairness
  4. Political Neutrality
  5. Accuracy--Fairness
  6. The Value of Inconsistent Metrics
  7. Inconsistent metrics encode the (unfortunately) inconsistent human values required in pluralistic alignment.
  8. Inconsistent metrics better preserves information of the underlying concept.
  9. Metric Inconsistency Has Potential Practical Values
  10. Mechanism I: Gradient Conflict as Implicit Regularization
  11. Mechanism II: The Existence of Pareto Fronts and the Rashomon Set
  12. Mechanism III: Complementary Metrics Block Shortcut Features
  13. Recommendations and Future Directions
  14. Advancing Practice-Driven Theories on Responsible AI.
  15. Documenting Normative Assumptions and Empirical Levels of Inconsistency.
  16. ...and 13 more sections

Figures (2)

  • Figure 1: An Overview of The Values in Theoretical Inconsistency
  • Figure 2: Represented as clusters in different colors, different groups have different ideas about what "fairness" means and formalized them into divergent metrics. By the Impossibility Theorem of Fairness, no models, as represented by black dots, can satisfy all of the three formalized metrics of fairness.