Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empowering Affected Individuals to Shape AI Fairness Assessments: Processes, Criteria, and Tools

Lin Luo, Satwik Ghanta, Yuri Nakao, Mathieu Chollet, Simone Stumpf

TL;DR

AI fairness assessments have traditionally been conducted by experts or regulators using predefined metrics, often neglecting the fairness notions of those affected by decisions. This study partners 18 decision subjects in a credit-rating scenario and uses an interactive prototype to ground lay fairness notions in model features and translate them into concrete, operational criteria. It documents a two-phase process (grounding notions and translating them into metrics) and reveals a diverse set of criteria across outcome and procedural fairness, including custom and combined metrics. The findings yield design implications for processes and tools to support inclusive, value-sensitive fairness assessment, demonstrating the feasibility and potential impact of stakeholder-driven fairness criteria in real-world AI systems.

Abstract

AI systems are increasingly used in high-stakes domains such as credit rating, where fairness concerns are critical. Existing fairness assessments are typically conducted by AI experts or regulators using predefined protected attributes and metrics, which often fail to capture the diversity and nuance of fairness notions held by the individuals who are affected by these systems' decisions, such as decision subjects. Recent work has therefore called for involving affected individuals in fairness assessment, yet little empirical evidence exists on how they create their own fairness criteria or what kinds of criteria they produce - knowledge that could not only inform experts' fairness evaluation and mitigation, but also guide the design of AI assessment tools. We address this gap through a qualitative user study with 18 participants in a credit rating scenario. Participants first articulated their fairness notions in their own words. Then, participants turned them into concrete quantified and operationalized fairness criteria, through an interactive prototype we designed. Our findings provide empirical evidence of the process through which people's fairness notions emerge via grounding in model features, and uncover a diverse set of individuals' custom-defined criteria for both outcome and procedural fairness. We provide design implications for processes and tools that support more inclusive and value-sensitive AI fairness assessment.

Empowering Affected Individuals to Shape AI Fairness Assessments: Processes, Criteria, and Tools

TL;DR

AI fairness assessments have traditionally been conducted by experts or regulators using predefined metrics, often neglecting the fairness notions of those affected by decisions. This study partners 18 decision subjects in a credit-rating scenario and uses an interactive prototype to ground lay fairness notions in model features and translate them into concrete, operational criteria. It documents a two-phase process (grounding notions and translating them into metrics) and reveals a diverse set of criteria across outcome and procedural fairness, including custom and combined metrics. The findings yield design implications for processes and tools to support inclusive, value-sensitive fairness assessment, demonstrating the feasibility and potential impact of stakeholder-driven fairness criteria in real-world AI systems.

Abstract

AI systems are increasingly used in high-stakes domains such as credit rating, where fairness concerns are critical. Existing fairness assessments are typically conducted by AI experts or regulators using predefined protected attributes and metrics, which often fail to capture the diversity and nuance of fairness notions held by the individuals who are affected by these systems' decisions, such as decision subjects. Recent work has therefore called for involving affected individuals in fairness assessment, yet little empirical evidence exists on how they create their own fairness criteria or what kinds of criteria they produce - knowledge that could not only inform experts' fairness evaluation and mitigation, but also guide the design of AI assessment tools. We address this gap through a qualitative user study with 18 participants in a credit rating scenario. Participants first articulated their fairness notions in their own words. Then, participants turned them into concrete quantified and operationalized fairness criteria, through an interactive prototype we designed. Our findings provide empirical evidence of the process through which people's fairness notions emerge via grounding in model features, and uncover a diverse set of individuals' custom-defined criteria for both outcome and procedural fairness. We provide design implications for processes and tools that support more inclusive and value-sensitive AI fairness assessment.
Paper Structure (34 sections, 3 equations, 4 figures, 3 tables)

This paper contains 34 sections, 3 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Assessing AI Fairness: Current Practice and Its Limitations
  4. Stakeholder Involvement in AI Fairness Assessment
  5. Methods
  6. Context Setup: AI Scenario and Participant Recruitment
  7. Session Procedure: Supporting Stakeholders Creating Fairness Criteria
  8. Prototype
  9. Data Collection and Analysis
  10. Results
  11. RQ1: How do stakeholders create their own fairness criteria?
  12. Phase 1: Grounding Fairness Notions Through Features
  13. Phase 2: Translating Notions into Operational Fairness Criteria
  14. RQ2: What kinds of fairness criteria do stakeholders create?
  15. Using existing outcome fairness metrics
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Module 1: Using Existing Metrics. Participants selected features (A), e.g., Gender, binning Gender into protected (Female) and non-protected (Male), and chose existing fairness metrics (B), e.g., Equal Opportunity. Module 3: Creating new outcome fairness metrics (C). This figure shows an example metric defined by a participant: P(AI Predicted = Good Credit | Ground Truth = Good Credit $\land$ Foreign Worker = Yes $\land$ Credit History = Delayed in paying off in the past). Module 4: Creating new procedural fairness metrics (D). This figure shows a procedural fairness criterion with 4 procedural rules: marking Gender as "Does Not Contribute"; setting Foreign Worker to have lower importance than Credit History using "Importance by Group"; setting the importance value of the Age feature to zero or lower by using "Feature Importance"; and indicating "no pre-checks" for the feature Purpose.
  • Figure 2: Actions with timeline for each participant in Phase 1: Grounding Fairness Notions Through Feature. The y-axis represents individual participants, with each row corresponding to one participant's creation process. Colored segments indicate themes associated with different actions involved in constructing fairness criteria. The x-axis represents the normalized progression of each participant's interaction session from start to finish (0–100%), derived from each participant's session transcript using Nvivo. The position of each colored segment reflects when a particular action (i.e., theme) occurred within the overall creation process; the length of the segment represents the relative duration for which that action was present during the session.
  • Figure 3: Different Ways of Criteria Created with Count by Participant
  • Figure 4: Metric Explanation Card: This figure illustrates the visualization and textual explanation of the Equalized Odds and Counterfactual Fairness metrics. Equalized Odds (composed of Equal Opportunity and Predictive Equality) represents a group fairness metric, while Counterfactual Fairness represents an individual fairness metric. All eight existing metrics follow a consistent explanation and visualization style.