Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Models

Hanze Guo, Jing Yao, Xiao Zhou, Xiaoyuan Yi, Xing Xie

TL;DR

This work tackles the problem of aligning LLMs to pluralistic human values by modeling interdependent value dimensions with priority scores using a structural causal model (SCM). It introduces COUPLE, a three-step inference-time framework that performs value attribution, value intervention, and counterfactual prediction to generate outputs aligned with target value profiles, while enhancing interpretability through explicit value concepts. The approach demonstrates improved accuracy, steerability, and interpretability across two value systems and both closed- and open-source LLMs, supported by comprehensive automatic and human evaluations, ablation studies, and case analyses. The work also discusses limitations, including dependence on large models, granularity of value representations, cross-cultural generalization, scalability, and evaluator bias, and suggests directions for future work.

Abstract

As large language models (LLMs) become increasingly integrated into applications serving users across diverse cultures, communities and demographics, it is critical to align LLMs with pluralistic human values beyond average principles (e.g., HHH). In psychological and social value theories such as Schwartz's Value Theory, pluralistic values are represented by multiple value dimensions paired with various priorities. However, existing methods encounter two challenges when aligning with such fine-grained value objectives: 1) they often treat multiple values as independent and equally important, ignoring their interdependence and relative priorities (value complexity); 2) they struggle to precisely control nuanced value priorities, especially those underrepresented ones (value steerability). To handle these challenges, we propose COUPLE, a COUnterfactual reasoning framework for PLuralistic valuE alignment. It introduces a structural causal model (SCM) to feature complex interdependency and prioritization among features, as well as the causal relationship between high-level value dimensions and behaviors. Moreover, it applies counterfactual reasoning to generate outputs aligned with any desired value objectives. Benefitting from explicit causal modeling, COUPLE also provides better interpretability. We evaluate COUPLE on two datasets with different value systems and demonstrate that COUPLE advances other baselines across diverse types of value objectives.

Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Models

TL;DR

This work tackles the problem of aligning LLMs to pluralistic human values by modeling interdependent value dimensions with priority scores using a structural causal model (SCM). It introduces COUPLE, a three-step inference-time framework that performs value attribution, value intervention, and counterfactual prediction to generate outputs aligned with target value profiles, while enhancing interpretability through explicit value concepts. The approach demonstrates improved accuracy, steerability, and interpretability across two value systems and both closed- and open-source LLMs, supported by comprehensive automatic and human evaluations, ablation studies, and case analyses. The work also discusses limitations, including dependence on large models, granularity of value representations, cross-cultural generalization, scalability, and evaluator bias, and suggests directions for future work.

Abstract

As large language models (LLMs) become increasingly integrated into applications serving users across diverse cultures, communities and demographics, it is critical to align LLMs with pluralistic human values beyond average principles (e.g., HHH). In psychological and social value theories such as Schwartz's Value Theory, pluralistic values are represented by multiple value dimensions paired with various priorities. However, existing methods encounter two challenges when aligning with such fine-grained value objectives: 1) they often treat multiple values as independent and equally important, ignoring their interdependence and relative priorities (value complexity); 2) they struggle to precisely control nuanced value priorities, especially those underrepresented ones (value steerability). To handle these challenges, we propose COUPLE, a COUnterfactual reasoning framework for PLuralistic valuE alignment. It introduces a structural causal model (SCM) to feature complex interdependency and prioritization among features, as well as the causal relationship between high-level value dimensions and behaviors. Moreover, it applies counterfactual reasoning to generate outputs aligned with any desired value objectives. Benefitting from explicit causal modeling, COUPLE also provides better interpretability. We evaluate COUPLE on two datasets with different value systems and demonstrate that COUPLE advances other baselines across diverse types of value objectives.
Paper Structure (74 sections, 6 equations, 8 figures, 29 tables, 1 algorithm)

This paper contains 74 sections, 6 equations, 8 figures, 29 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pluralistic Value Alignment
  4. Causal Tasks in Large Language Models
  5. Preliminary
  6. Task Definitions
  7. Causal Theory Foundation
  8. Methodology
  9. The Whole Framework
  10. Value Attribution
  11. Criteria Calibration.
  12. Counterfactual Prediction
  13. Experiments
  14. Experimental Settings
  15. Datasets and Evaluation
  16. ...and 59 more sections

Figures (8)

  • Figure 1: Illustration of pluralistic human values. Two users prioritize value dimensions (self-direction, benevolence, and security) differently, leading to divergent judgments on the same question.
  • Figure 2: Illustration of the COUPLE framework, with a three-step counterfactual workflow.
  • Figure 3: Human evaluation results of COUPLE against baselines
  • Figure 4: Analysis for the advantages of COUPLE. (a) Performance under different numbers of value dimensions; (b) Performance across varying deviations toward the alignment objective; (c), (d) MAE deviation when shifting a value from the original priority to the target priority under Plan and Solve, and COUPLE; smaller values indicate better alignment.
  • Figure 5: Case study on fine-grained pluralistic value alignment. COUPLE provides fine-grained value alignment while maintaining interpretability.
  • ...and 3 more figures