The Value of Disagreement in AI Design, Evaluation, and Alignment

TL;DR

The paper tackles perspectival homogenization in AI design, evaluation, and alignment by introducing a normative framework that values disagreement. It grounds its approach in four epistemic rationales—diversity benefits, standpoint epistemology, productive disagreement, and higher-order evidence—and organizes interventions around antecedent, process, and outcomes stages of AI workflows. By linking design decisions to these rationales, it demonstrates how disagreement can improve accuracy, fairness, and robustness, while outlining concrete practices for inclusive participation, networked collaboration, and transparent documentation. The work provides a principled foundation for participatory and pluralistic AI governance and charts actionable avenues for future research and governance mechanisms.

Abstract

Disagreements are widespread across the design, evaluation, and alignment pipelines of artificial intelligence (AI) systems. Yet, standard practices in AI development often obscure or eliminate disagreement, resulting in an engineered homogenization that can be epistemically and ethically harmful, particularly for marginalized groups. In this paper, we characterize this risk, and develop a normative framework to guide practical reasoning about disagreement in the AI lifecycle. Our contributions are two-fold. First, we introduce the notion of perspectival homogenization, characterizing it as a coupled ethical-epistemic risk that arises when an aspect of an AI system's development unjustifiably suppresses disagreement and diversity of perspectives. We argue that perspectival homogenization is best understood as a procedural risk, which calls for targeted interventions throughout the AI development pipeline. Second, we propose a normative framework to guide such interventions, grounded in lines of research that explain why disagreement can be epistemically beneficial, and how its benefits can be realized in practice. We apply this framework to key design questions across three stages of AI development tasks: when disagreement is epistemically valuable; whose perspectives should be included and preserved; how to structure tasks and navigate trade-offs; and how disagreement should be documented and communicated. In doing so, we challenge common assumptions in AI practice, offer a principled foundation for emerging participatory and pluralistic approaches, and identify actionable pathways for future work in AI design and governance.

Figures (1)

• Figure 1: A schematic depiction of the proposed framework for reasoning about how disagreement at different stages of AI development tasks---such as data labeling and annotation, red-teaming, ruleset elicitation for value alignment---wherein divergences in opinions or perspective may be expected. In all such tasks, researchers and practitioners face challenging questions at different stages of tasks about potential epistemic value of disagreements, how best to proceed in a given stage to realize those values (Figure inspired by Figure 1 in huang2024collective.)