Value Profiles for Encoding Human Variation

TL;DR

The paper addresses encoding human annotator variation in subjective tasks by introducing natural language value profiles (V) and a steerable decoder that conditions predictions on V. It develops an encoder–decoder framework with a value profile encoder and a decoder, and introduces an information-theoretic methodology based on $V$-information to compare representations. Key findings show that demonstrations provide the most information, value profiles offer substantial predictive power and compress over 70% of usable information from demonstrations, and demographics contribute little; clustering value profiles yields more informative groupings than demographics. Extrinsically, the value-profile decoder is interpretable and well calibrated, can simulate an annotator population, and supports pluralistic AI alignment. Overall, the work provides a scalable, interpretable approach to describing individual variation beyond demographic groups, with implications for personalization, safety, and fairness.

Abstract

Modelling human variation in rating tasks is crucial for personalization, pluralistic model alignment, and computational social science. We propose representing individuals using natural language value profiles -- descriptions of underlying values compressed from in-context demonstrations -- along with a steerable decoder model that estimates individual ratings from a rater representation. To measure the predictive information in a rater representation, we introduce an information-theoretic methodology and find that demonstrations contain the most information, followed by value profiles, then demographics. However, value profiles effectively compress the useful information from demonstrations (>70% information preservation) and offer advantages in terms of scrutability, interpretability, and steerability. Furthermore, clustering value profiles to identify similarly behaving individuals better explains rater variation than the most predictive demographic groupings. Going beyond test set performance, we show that the decoder predictions change in line with semantic profile differences, are well-calibrated, and can help explain instance-level disagreement by simulating an annotator population. These results demonstrate that value profiles offer novel, predictive ways to describe individual variation beyond demographics or group information.

Figures (20)

• Figure 1: The value profile autoencoder setup. Decoder outputs are from trained profile decoder while demographics are illustrative to preserve privacy. The encoder extracts/compresses value information from rater examples, and the decoder changes predictions on held-out questions according to the value profile.
• Figure 2: Rater representations and example corresponding decoder prompts ($\emptyset$, $D$, $V$, $E_n$). The decoder predicts the rater's annotation given the rater representation.
• Figure 3: Test losses across rater representation settings. Dashed line: label entropy $H(\mathcal{Y})$; no info: $\emptyset$; profile*: value profiles $V$ generated by gemma2-{9/27}b / Gemini-1.5-Pro; dem (all): $D$; N ex: $E_N$, up to $N$ examples from $D_i^\textrm{fit}$. ValuePrism does not have demographics, but does have a ground truth value profile. Each dot corresponds to a run with a differently seeded train/test split, with 95% CI reported. Generally, in-context examples are more performant than value profiles, which are more performant than demographics.
• Figure 4: Usable rater information across datasets and rater representations (95% CI).
• Figure 5: Info. preserved w.r.t. to using all examples. Results shown on the four large, low-variance datasets. Gemini profiles preserve $>$70% of usable information.