Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment

Andrew Konya, Aviv Ovadya, Kevin Feng, Quan Ze Chen, Lisa Schirch, Colin Irwin, Amy X. Zhang

TL;DR

A method to measure the alignment between public will and language model (LM) behavior that can be applied to fine-tuning, online oversight, and pre-release safety checks and is validated by applying across three different domains of LM prompts related to mental health.

Abstract

We introduce a method to measure the alignment between public will and language model (LM) behavior that can be applied to fine-tuning, online oversight, and pre-release safety checks. Our `chain of alignment' (CoA) approach produces a rule based reward (RBR) by creating model behavior $\textit{rules}$ aligned to normative $\textit{objectives}$ aligned to $\textit{public will}$. This factoring enables a nonexpert public to directly specify their will through the normative objectives, while expert intelligence is used to figure out rules entailing model behavior that best achieves those objectives. We validate our approach by applying it across three different domains of LM prompts related to mental health. We demonstrate a public input process built on collective dialogues and bridging-based ranking that reliably produces normative objectives supported by at least $96\% \pm 2\%$ of the US public. We then show that rules developed by mental health experts to achieve those objectives enable a RBR that evaluates an LM response's alignment with the objectives similarly to human experts (Pearson's $r=0.841$, $AUC=0.964$). By measuring alignment with objectives that have near unanimous public support, these CoA RBRs provide an approximate measure of alignment between LM behavior and public will.

Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment

TL;DR

A method to measure the alignment between public will and language model (LM) behavior that can be applied to fine-tuning, online oversight, and pre-release safety checks and is validated by applying across three different domains of LM prompts related to mental health.

Abstract

We introduce a method to measure the alignment between public will and language model (LM) behavior that can be applied to fine-tuning, online oversight, and pre-release safety checks. Our `chain of alignment' (CoA) approach produces a rule based reward (RBR) by creating model behavior aligned to normative aligned to . This factoring enables a nonexpert public to directly specify their will through the normative objectives, while expert intelligence is used to figure out rules entailing model behavior that best achieves those objectives. We validate our approach by applying it across three different domains of LM prompts related to mental health. We demonstrate a public input process built on collective dialogues and bridging-based ranking that reliably produces normative objectives supported by at least of the US public. We then show that rules developed by mental health experts to achieve those objectives enable a RBR that evaluates an LM response's alignment with the objectives similarly to human experts (Pearson's , ). By measuring alignment with objectives that have near unanimous public support, these CoA RBRs provide an approximate measure of alignment between LM behavior and public will.

Paper Structure

This paper contains 18 sections, 25 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Experiments
  3. Creating objectives aligned with public will
  4. Creating rules aligned with objectives
  5. Measuring alignment via rules
  6. Implications, Limitations, and Future Work
  7. Appendix
  8. Mental Health Subdomains
  9. Normative objective creation process details
  10. Generating normative objectives v0 via public input
  11. Refining the normative objectives
  12. Public vote and ratification of the normative objectives
  13. Examples
  14. Participant distribution
  15. Testing the effect of objectives-rule alignment weights in the RBR via ablation
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Our approach produces objectives and rules that form a "chain of alignment" linking model behavior to public will (bottom). We test our approach across three domains of LM behavior, and evaluate each link in the resulting alignment chain (top): A) Public support for the objectives gives a measure of their alignment with public will. B) The distribution of rules' alignment with the objectives is produced by domain experts assessing each rule's liklelihood to help achieve the objectives. C) The rules' ability to measure if model behavior aligns with the objectives is assessed by comparing the output of an LM-graded rule based reward (x-axis) with domain expert assessments of alignment with objectives (y-axis) for a diverse sample of {user prompt, LM response} pairs.
  • Figure 2: Diagram of process for creating normative objectives.
  • Figure 3: Distribution of our sample relative to benchmarks for the adult US public.
  • Figure 4: Rule-objective alignment evaluation performance compared to human experts. Plotted values computed by averaging Dice-Hamming similarities between evaluator outputs and the judgements of multiple individual human experts, then normalizing those values so that average similarity between human experts is one.