PERSONA: A Reproducible Testbed for Pluralistic Alignment

TL;DR

This paper tackles the challenge of aligning language models with a diverse spectrum of user values by introducing PERSONA, a reproducible testbed that uses 1,586 synthetic personas derived from US census data and a large-scale prompt/feedback dataset to evaluate pluralistic alignment. It provides a framework for role-playing diverse users, human-verified evaluation, and the creation of the PERSONA Bench to benchmark alignment methods. Key findings include the effectiveness of LLMs as synthetic evaluators, the nuanced benefits of persona summarization over chain-of-thought reasoning, and insights from extensive human evaluations. The work offers a scalable, demographically grounded platform to develop and assess personalized and pluralistic alignment approaches, while noting limitations related to demographic scope and synthetic data realism.

Abstract

The rapid advancement of language models (LMs) necessitates robust alignment with diverse user values. However, current preference optimization approaches often fail to capture the plurality of user opinions, instead reinforcing majority viewpoints and marginalizing minority perspectives. We introduce PERSONA, a reproducible test bed designed to evaluate and improve pluralistic alignment of LMs. We procedurally generate diverse user profiles from US census data, resulting in 1,586 synthetic personas with varied demographic and idiosyncratic attributes. We then generate a large-scale evaluation dataset containing 3,868 prompts and 317,200 feedback pairs obtained from our synthetic personas. Leveraging this dataset, we systematically evaluate LM capabilities in role-playing diverse users, verified through human judges, and the establishment of both a benchmark, PERSONA Bench, for pluralistic alignment approaches as well as an extensive dataset to create new and future benchmarks. The full dataset and benchmarks are available here: https://www.synthlabs.ai/research/persona.

Figures (13)

• Figure 1: Procedure for generating personas. The above is a flow graph outlining the generation of a single persona. An exact example for this generation process can be found in the appendix. First, we sample a subset of US census data and query a language model to see if the resulting persona is self consistent. If it isn't, we resample. Next, we use procedural methods to fill in missing components of the census data. The list of procedural methods can be found in the appendix. Finally, we use a language model to fill in open ended psychoanalytic attributes.
• Figure 2: Histograms of group statistics of our demographic of synthetic personas.
• Figure 3: Distribution of prompt topics in the Persona dataset. The prompts are taken from kirk2024prism, and any differences in the distribution are due to filtering and difference in topics clustering.
• Figure 4: High level for going from the original PRISM dataset to a confusion matrix of Cohen's Kappa between simulated personas. The robot emoji signifies the inclusion of a language model, where as the person emoji signifies the use of a persona (or multiple.)
• Figure 5: Leave one out analysis of various attributes of our persona. Influence is measured as the annotator agreement (Cohen's kappa) between an annotator with a given attribute and an annotator without said attribute. Lower Cohen's kappa equates to larger influence.