Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowing But Not Doing: Convergent Morality and Divergent Action in LLMs

Jen-tse Huang, Jiantong Qin, Xueli Qiu, Sharon Levy, Michelle R. Kaufman, Mark Dredze

TL;DR

This work tackles how LLMs represent and enact human values by pairing self-reported values (PVQ-40) with enacted choices in ValAct-15k, a 3,000-scenario benchmark drawn from Reddit and grounded in Schwartz's ten basic values. It evaluates ten frontier LLMs and 55 human participants across five domains, revealing near-perfect cross-model convergence in value-enactment ($r \approx 1.0$) but substantial human variability and a systematic knowledge–action gap ($r \approx 0.3\-0.4$). A second experiment shows role-play resistance, with value-adoption prompts reducing performance by up to $6.6\%$ (and up to $10\%$ in constrained cases), indicating limits of instruction-tuning for stable value-conditioned behavior. The findings imply that current alignment pipelines yield normative but homogeneous value profiles and that scenario-based behavioral benchmarks are essential to capture enacted values, with important implications for safety, governance, and the deployment of AI in value-laden domains.

Abstract

Value alignment is central to the development of safe and socially compatible artificial intelligence. However, how Large Language Models (LLMs) represent and enact human values in real-world decision contexts remains under-explored. We present ValAct-15k, a dataset of 3,000 advice-seeking scenarios derived from Reddit, designed to elicit ten values defined by Schwartz Theory of Basic Human Values. Using both the scenario-based questions and the traditional value questionnaire, we evaluate ten frontier LLMs (five from U.S. companies, five from Chinese ones) and human participants ($n = 55$). We find near-perfect cross-model consistency in scenario-based decisions (Pearson $r \approx 1.0$), contrasting sharply with the broad variability observed among humans ($r \in [-0.79, 0.98]$). Yet, both humans and LLMs show weak correspondence between self-reported and enacted values ($r = 0.4, 0.3$), revealing a systematic knowledge-action gap. When instructed to "hold" a specific value, LLMs' performance declines up to $6.6%$ compared to merely selecting the value, indicating a role-play aversion. These findings suggest that while alignment training yields normative value convergence, it does not eliminate the human-like incoherence between knowing and acting upon values.

Knowing But Not Doing: Convergent Morality and Divergent Action in LLMs

TL;DR

This work tackles how LLMs represent and enact human values by pairing self-reported values (PVQ-40) with enacted choices in ValAct-15k, a 3,000-scenario benchmark drawn from Reddit and grounded in Schwartz's ten basic values. It evaluates ten frontier LLMs and 55 human participants across five domains, revealing near-perfect cross-model convergence in value-enactment () but substantial human variability and a systematic knowledge–action gap (). A second experiment shows role-play resistance, with value-adoption prompts reducing performance by up to (and up to in constrained cases), indicating limits of instruction-tuning for stable value-conditioned behavior. The findings imply that current alignment pipelines yield normative but homogeneous value profiles and that scenario-based behavioral benchmarks are essential to capture enacted values, with important implications for safety, governance, and the deployment of AI in value-laden domains.

Abstract

Value alignment is central to the development of safe and socially compatible artificial intelligence. However, how Large Language Models (LLMs) represent and enact human values in real-world decision contexts remains under-explored. We present ValAct-15k, a dataset of 3,000 advice-seeking scenarios derived from Reddit, designed to elicit ten values defined by Schwartz Theory of Basic Human Values. Using both the scenario-based questions and the traditional value questionnaire, we evaluate ten frontier LLMs (five from U.S. companies, five from Chinese ones) and human participants (). We find near-perfect cross-model consistency in scenario-based decisions (Pearson ), contrasting sharply with the broad variability observed among humans (). Yet, both humans and LLMs show weak correspondence between self-reported and enacted values (), revealing a systematic knowledge-action gap. When instructed to "hold" a specific value, LLMs' performance declines up to compared to merely selecting the value, indicating a role-play aversion. These findings suggest that while alignment training yields normative value convergence, it does not eliminate the human-like incoherence between knowing and acting upon values.
Paper Structure (36 sections, 3 equations, 7 figures)

This paper contains 36 sections, 3 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Reddit posts collection.
  4. Human-in-the-loop LLM generation of value-based actions.
  5. LLM selections and hyper-parameters.
  6. Human study design.
  7. Participant recruitment.
  8. Evaluation metrics.
  9. Results
  10. Experiment 1: Value Measurement
  11. Cross-model convergence in value-based decisions.
  12. Human–human variability in values.
  13. Human–LLM convergence.
  14. The Knowledge–Action Gap: weak alignment between declared and enacted values.
  15. Experiment 2: Value Assignment
  16. ...and 21 more sections

Figures (7)

  • Figure 1: The pipeline to construct ValAct-15k. (a) The amount of posts we collected from Reddit and we finally selected. (b) The histogram of the number of tokens in each post. (c) The histogram of the number of tokens in each action generated by GPT or Qwen. (d) The histogram of the number of questions having actions generated by (GPT, Qwen) among all 15,000 questions. (e) The frequency of four LLMs choosing actions generated by GPT or Qwen under three scenarios.
  • Figure 2: Basic statistics of our human evaluation. Figure (a) counts all 55 responses while the remaining figures use the 47 responses whose attention check is valid. The Cohen's kappa in Figure (b) assumes the performance by chance is $\frac{1}{4}$.
  • Figure 3: The comparison of LLM and human results from PVQ-40 and ValAct-15k. The error bars show $\pm95\%$ confidence levels.
  • Figure 4: Pearson correlations between different settings using the ten-dimension value results.
  • Figure 5: The histogram of Pearson correlations between humans using PVQ-40 and ValAct-15k.
  • ...and 2 more figures