Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values
Henry Bell, Lara Neubauer da Costa Schertel, Bochu Ding, Brandon Fain
TL;DR
The paper introduces Grounded Constitutional AI (GCAI), a framework that learns constitutions of contextual and general principles by grounding them in human reasons and values, extending Inverse Constitutional AI (ICAI). It uses a four-stage elicitation pipeline—candidate generation, clustering, summarization, and scoring/selection—applied to contextual data from HelpSteer2 and general data from PRISM, with hierarchical and proportionally fair clustering to form representative principles, and selects a final constitution with top principles from each source at $K=10$. Through large-scale human surveys and model benchmarking, GCAI constitutions are shown to be preferred over ICAI across multiple dimensions (moral grounding, coherence, consensus), and the aligned models exhibit stronger emphasis on safety, ethics, and fairness, while maintaining comparable factual and domain knowledge. The work discusses the interpretability, transparency, and ratification requirements for constitutions in AI alignment, and highlights the role of justifications in increasing faithfulness and public justification, while acknowledging limitations and the need for stakeholder ratification. Overall, GCAI provides a scalable pathway to generate more morally grounded, coherent, and pluralistic alignment principles for AI systems, with practical implications for governance and deployment draws on normative theory and empirical evaluation.
Abstract
A crucial consideration when developing and deploying Large Language Models (LLMs) is the human values to which these models are aligned. In the constitutional framework of alignment models are aligned to a set of principles (the constitution) specified in natural language. However, it is unclear how to fairly determine this constitution with widespread stakeholder input. In this work we propose Grounded Constitutional AI (GCAI), a unified framework for generating constitutions of principles that are representative of both users' general expectations toward AI (general principles) and their interaction-time preferences (contextual principles). We extend the Inverse Constitutional AI (ICAI) approach to generate contextual principles from human preference annotation data by leveraging human-provided \textit{reasons} for their preferences. We supplement these contextual principles with general principles surfaced from user statements of \textit{values} regarding AI. We show that a constitution generated by GCAI is preferred by humans over one generated through ICAI both personally, and for widespread use in governing AI behavior. Additionally participants consider the GCAI constitution to be more morally grounded, coherent, and pluralistic.