Policy Prototyping for LLMs: Pluralistic Alignment via Interactive and Collaborative Policymaking

TL;DR

The paper tackles how to broaden stakeholder input in LLM policy formation and closes the loop between input and model behavior by introducing policy prototyping. It proposes an interactive, design-prototyping inspired process that enables collaborative drafting, testing, and refinement of policies with policy-informed models. Findings from a 15-week industrial lab study yield four guiding principles emphasizing direct experimentation, synchronous collaboration, low-fidelity prototyping, and scenario-based artifacts. The work argues that policy prototyping complements existing alignment methods and expands the methodological repertoire for pluralistic alignment, with practical implications for cost, scalability, and tooling.

Abstract

Emerging efforts in AI alignment seek to broaden participation in shaping model behavior by eliciting and integrating collective input into a policy for model finetuning. While pluralistic, these processes are often linear and do not allow participating stakeholders to confirm whether potential outcomes of their contributions are indeed consistent with their intentions. Design prototyping has long advocated for rapid iteration using tight feedback loops of ideation, experimentation, and evaluation to mitigate these issues. We thus propose policy prototyping for LLMs, a new process that draws inspiration from prototyping practices to enable stakeholders to collaboratively and interactively draft LLM policies. Through learnings from a real-world LLM policymaking initiative at an industrial AI lab, we motivate our approach and characterize policy prototyping with four guiding principles. Because policy prototyping emphasizes a contrasting set of priorities compared to previous approaches, we envision our approach to be a valuable addition to the methodological repertoire for collaborative, pluralistic alignment.

Figures (1)

• Figure 1: An existing process for collective constitutional input via Collective Constitutional AI huang2024collective (top) alongside our proposed policy prototyping process (bottom). Our process can complement existing approaches and broaden pluralistic alignment's methodological repertoire.