Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferences

TL;DR

The paper introduces the Deep Value Benchmark (DVB) to quantify whether large language models generalize deep human values or merely shallow preferences. It uses a confound-then-deconfound design with training data where deep values co-vary with shallow features and testing data where these cues are decoupled, yielding a Deep Value Generalization Rate (DVGR). Across nine models, DVGR averages about 0.30, with all results below chance, indicating a strong bias toward shallow preference generalization; explicit value guidance improves DVGR modestly, while scaling does not fix the issue. The authors release a validated dataset and present a general framework for measuring alignment-relevant generalization, offering a roadmap for future benchmarks and methods to promote deeper value generalization in AI systems.

Abstract

We introduce the Deep Value Benchmark (DVB), an evaluation framework that directly tests whether large language models (LLMs) learn fundamental human values or merely surface-level preferences. This distinction is critical for AI alignment: Systems that capture deeper values are likely to generalize human intentions robustly, while those that capture only superficial patterns in preference data risk producing misaligned behavior. The DVB uses a novel experimental design with controlled confounding between deep values (e.g., moral principles) and shallow features (e.g., superficial attributes). In the training phase, we expose LLMs to human preference data with deliberately correlated deep and shallow features -- for instance, where a user consistently prefers (non-maleficence, formal language) options over (justice, informal language) alternatives. The testing phase then breaks these correlations, presenting choices between (justice, formal language) and (non-maleficence, informal language) options. This design allows us to precisely measure a model's Deep Value Generalization Rate (DVGR) -- the probability of generalizing based on the underlying value rather than the shallow feature. Across 9 different models, the average DVGR is just 0.30. All models generalize deep values less than chance. Larger models have a (slightly) lower DVGR than smaller models. We are releasing our dataset, which was subject to three separate human validation experiments. DVB provides an interpretable measure of a core feature of alignment.

Figures (8)

• Figure 1: Conceptual overview of confound-then-deconfound design.
• Figure 2: Example of a generated choice completion for a tuple where $\textcolor{valueonecolor}{v_1}$ is non-maleficence ("An Agent should not harm others") and $\textcolor{valueonecolor}{s_1}$ is adaptive ("Preferring AI agents adjust their behavior based on user interactions"); $\textcolor{valuetwocolor}{v_2}$ is reparation ("An Agent should correct past errors") and $\textcolor{valuetwocolor}{s_2}$ is static ("Preferring AI agents maintain consistent behavior regardless of user interactions"). This occurs in a legal context. The O*NET activity is "analyzing data or information".
• Figure 3: Test template that models saw. Each test question is administered as its own prompt.
• Figure 4: Experiment results. For (a) and (b), error bars are 95% CIs using the Wilson method wilson_probable_1927.
• Figure 5: The prompt that turned templates into natural language choices between two options. \ref{['choice-completion']} gives an example of a completion from this template.