Controllable Value Alignment in Large Language Models through Neuron-Level Editing

TL;DR

This work identifies a fundamental challenge in aligning LLMs to human values: steering a target value often leaks into non-target values due to entangled representations. It introduces value leakage as a diagnostic and presents two metrics, NLR and NGLR, to quantify magnitude and structural propagation of leakage. To mitigate leakage, the authors propose NeVA, a neuron-level editing framework that localizes interventions to sparse, value-relevant neurons identified via value-specific probes and context-aware editing rules, enabling strong target-value control while preserving general capabilities. Across multiple backbones and datasets, NeVA achieves superior control (higher CSR) with minimal degradation in fluency and reasoning (MMLU), and exhibits significantly reduced leakage (lower NLR) with leakage largely contained within semantically related value groups. The approach offers a more controllable, interpretable mechanism for value alignment with potential applicability to user-specific or scenario-specific value customization, while acknowledging limitations and risks related to value theory assumptions and potential misuse.

Abstract

Aligning large language models (LLMs) with human values has become increasingly important as their influence on human behavior and decision-making expands. However, existing steering-based alignment methods suffer from limited controllability: steering a target value often unintentionally activates other, non-target values. To characterize this limitation, we introduce value leakage, a diagnostic notion that captures the unintended activation of non-target values during value steering, along with a normalized leakage metric grounded in Schwartz's value theory. In light of this analysis, we propose NeVA, a neuron-level editing framework for controllable value alignment in LLMs. NeVA identifies sparse, value-relevant neurons and performs inference-time activation editing, enabling fine-grained control without parameter updates or retraining. Experiments show that NeVA achieves stronger target value alignment while incurring smaller performance degradation on general capability. Moreover, NeVA significantly reduces the average leakage, with residual effects largely confined to semantically related value classes. Overall, NeVA offers a more controllable and interpretable mechanism for value alignment.

Figures (12)

• Figure 1: Illustration of value leakage in the steering-based alignment. The radar chart presents the 10 basic Schwartz value alignment results under four different target value steering interventions. Compared with the base LLaMA-3-8B results (gray line), we can find that steering a single target value usually induces non-target value activations.
• Figure 2: From global steering to localized control: NeVA formulates value alignment as localized, neuron-level intervention. The framework decomposes dense value representations into sparse, value-relevant neurons and selectively modulates their contributions during generation. This shift in abstraction enables fine-grained controllability and mitigates value leakage across non-target dimensions.
• Figure 3: Target value alignment results on LLaMA-3-8B-Instruct. Each line represents a value alignment method. Results on LLaMA-2-13B-Chat and Qwen2.5-7B-Instruct are reported in Appendix \ref{['sec: appendix_backbones']}.
• Figure 4: Heatmaps of the Normanized Group Leakage Ratio (NGLR) of ConVA and NeVA.
• Figure 5: CSR comparison between NeVA-selected neurons and randomly selected neurons.