VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models

TL;DR

VALUEFLOW presents a unified end-to-end framework for value-based alignment of LLMs, integrating a hierarchical value embedding space (HiVES), a large-scale value-intensity database (VIDB) built on calibrated intensity via Plackett–Luce rankings, and a ranking-based evaluator for stable intensity scoring. It introduces cross-theory anchors to unify diverse value theories, and a two-stage training pipeline to align intra-theory and inter-theory representations. The framework enables steerable generation conditioned on (value, intensity) pairs and demonstrates controlled, multi-value steering with asymmetric dose–response patterns across models and values, plus demographic profiling and safety analyses. The approach provides scalable, reproducible audits for pluralistic, policy-steerable alignment and facilitates profiling, personalisation, and cross-cultural deployment of value-oriented LLM behavior. Overall, VALUEFLOW advances principled, scalable, and interpretable value-alignment tooling by coupling hierarchical representations, calibrated intensity evaluation, and end-to-end steering mechanisms.

Abstract

Aligning Large Language Models (LLMs) with the diverse spectrum of human values remains a central challenge: preference-based methods often fail to capture deeper motivational principles. Value-based approaches offer a more principled path, yet three gaps persist: extraction often ignores hierarchical structure, evaluation detects presence but not calibrated intensity, and the steerability of LLMs at controlled intensities remains insufficiently understood. To address these limitations, we introduce VALUEFLOW, the first unified framework that spans extraction, evaluation, and steering with calibrated intensity control. The framework integrates three components: (i) HIVES, a hierarchical value embedding space that captures intra- and cross-theory value structure; (ii) the Value Intensity DataBase (VIDB), a large-scale resource of value-labeled texts with intensity estimates derived from ranking-based aggregation; and (iii) an anchor-based evaluator that produces consistent intensity scores for model outputs by ranking them against VIDB panels. Using VALUEFLOW, we conduct a comprehensive large-scale study across ten models and four value theories, identifying asymmetries in steerability and composition laws for multi-value control. This paper establishes a scalable infrastructure for evaluating and controlling value intensity, advancing pluralistic alignment of LLMs.

Figures (42)

• Figure 1: Example of VALUEFLOW. An end-to-end framework that extracts value profiles via a hierarchical embedding model (HiVES), steers generation toward target value and intensity, and evaluates responses by ranking them against anchors in the Value Intensity DB (VIDB).
• Figure 2: Ratings across models. For the same items and values, models produce scores ranging from strong negative to positive.
• Figure 3: Overview of our framework: (a) construction of the Value Intensity DB (VIDB); (b) ranking-based evaluation that yields calibrated intensity scores. The VIDB built in (a) serves as the reference anchor set used in (b) to infer intensity via relative ranking.
• Figure 4: Steerability by model.Top: intensity-anchor prompts; bottom: user-text prompts. Bars show mean shift $\Delta \;=\; s_{\text{steered}} \;-\; s_{\text{default}}$. We underline one exemplar model that is visualized.
• Figure 5: HiVES vs. baselines. We report hierarchical ranking accuracy, similarity correlation, and disentanglement for theories.