Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Disclosure and Evaluation as Fairness Interventions for General-Purpose AI

Vyoma Raman, Judy Hanwen Shen, Andy K. Zhang, Lindsey Gailmard, Rishi Bommasani, Daniel E. Ho, Angelina Wang

TL;DR

General-purpose AI raises fairness concerns across unforeseen deployment contexts, making outcome-based fairness prescriptions impractical. The authors propose a process-focused approach centered on information gathering, disclosure, and multi-level evaluation of fairness across model, system, and society levels, with explicit provider and deployer obligations. They discuss regulatory alignment, supply-chain transparency, and a multi-dimensional evaluation framework (pre/post deployment, naturalistic/synthetic data, internal/external evaluators) to surface and address harms. The goal is context-sensitive, adaptive governance that scales with GPAI evolution while avoiding undue burdens on smaller actors.

Abstract

Despite conflicting definitions and conceptions of fairness, AI fairness researchers broadly agree that fairness is context-specific. However, when faced with general-purpose AI, which by definition serves a range of contexts, how should we think about fairness? We argue that while we cannot be prescriptive about what constitutes fair outcomes, we can specify the processes that different stakeholders should follow in service of fairness. Specifically, we consider the obligations of two major groups: system providers and system deployers. While system providers are natural candidates for regulatory attention, the current state of AI understanding offers limited insight into how upstream factors translate into downstream fairness impacts. Thus, we recommend that providers invest in evaluative research studying how model development decisions influence fairness and disclose whom they are serving their models to, or at the very least, reveal sufficient information for external researchers to conduct such research. On the other hand, system deployers are closer to real-world contexts and can leverage their proximity to end users to address fairness harms in different ways. Here, we argue they should responsibly disclose information about users and personalization and conduct rigorous evaluations across different levels of fairness. Overall, instead of focusing on enforcing fairness outcomes, we prioritize intentional information-gathering by system providers and deployers that can facilitate later context-aware action. This allows us to be specific and concrete about the processes even while the contexts remain unknown. Ultimately, this approach can sharpen how we distribute fairness responsibilities and inform more fluid, context-sensitive interventions as AI continues to advance.

Disclosure and Evaluation as Fairness Interventions for General-Purpose AI

TL;DR

General-purpose AI raises fairness concerns across unforeseen deployment contexts, making outcome-based fairness prescriptions impractical. The authors propose a process-focused approach centered on information gathering, disclosure, and multi-level evaluation of fairness across model, system, and society levels, with explicit provider and deployer obligations. They discuss regulatory alignment, supply-chain transparency, and a multi-dimensional evaluation framework (pre/post deployment, naturalistic/synthetic data, internal/external evaluators) to surface and address harms. The goal is context-sensitive, adaptive governance that scales with GPAI evolution while avoiding undue burdens on smaller actors.

Abstract

Despite conflicting definitions and conceptions of fairness, AI fairness researchers broadly agree that fairness is context-specific. However, when faced with general-purpose AI, which by definition serves a range of contexts, how should we think about fairness? We argue that while we cannot be prescriptive about what constitutes fair outcomes, we can specify the processes that different stakeholders should follow in service of fairness. Specifically, we consider the obligations of two major groups: system providers and system deployers. While system providers are natural candidates for regulatory attention, the current state of AI understanding offers limited insight into how upstream factors translate into downstream fairness impacts. Thus, we recommend that providers invest in evaluative research studying how model development decisions influence fairness and disclose whom they are serving their models to, or at the very least, reveal sufficient information for external researchers to conduct such research. On the other hand, system deployers are closer to real-world contexts and can leverage their proximity to end users to address fairness harms in different ways. Here, we argue they should responsibly disclose information about users and personalization and conduct rigorous evaluations across different levels of fairness. Overall, instead of focusing on enforcing fairness outcomes, we prioritize intentional information-gathering by system providers and deployers that can facilitate later context-aware action. This allows us to be specific and concrete about the processes even while the contexts remain unknown. Ultimately, this approach can sharpen how we distribute fairness responsibilities and inform more fluid, context-sensitive interventions as AI continues to advance.

Paper Structure

This paper contains 27 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Our Work
  3. Expanding the Scope of Fairness Interventions
  4. Levels of Fairness
  5. Model Level.
  6. System Level.
  7. Society Level.
  8. Actors and Priorities in AI Regulation
  9. Information-Gathering as a Fairness Intervention
  10. Provider Obligations
  11. Evaluating the Impact of Development Decisions.
  12. Disclosing Supply Chain Relationships.
  13. Deployer Obligations
  14. Disclosing Usage Contexts
  15. Social Group Labels of Users.
  16. ...and 12 more sections

Figures (1)

  • Figure 1: We recommend fairness-related obligations involving information-gathering for providers and deployers to help us analyze and mitigate harms at the model, system, and society level.