Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors

TL;DR

This work tackles the challenge of aligning LLMs to be both helpful and harmless by introducing Preference Vector, a modular framework that trains separate models for individual preferences and composes their effects at inference via parameter-space vectors. By extracting preference vectors as differences between opposite-preference models ($\phi_{\text{Helpful}} = \theta_{\text{Helpful+}} - \theta_{\text{Helpful-}}$ and $\phi_{\text{Harmless}} = \theta_{\text{Harmless+}} - \theta_{\text{Harmless-}}$) and aggregating into a base model with scalars ($\theta_{\text{Aggregated}} = \theta_{\text{Base}} + \eta_{\text{Helpful}} \cdot \phi_{\text{Helpful}} + \eta_{\text{Harmless}} \cdot \phi_{\text{Harmless}}$), the approach enables flexible, test-time control and easy extension to new preferences. Experiments on PKU-SafeRLHF across multiple models show improved helpfulness with comparable harmlessness, and demonstrate robustness of the vectors across seeds with a predominantly unidirectional structure. The results indicate that modular preference vectors can be added or scaled at inference to tailor behavior without retraining, offering a scalable path for multi-objective safety alignment in LLMs.

Abstract

Ensuring that large language models (LLMs) are both helpful and harmless is a critical challenge, as overly strict constraints can lead to excessive refusals, while permissive models risk generating harmful content. Existing approaches, such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO), attempt to balance these trade-offs but suffer from performance conflicts, limited controllability, and poor extendability. To address these issues, we propose Preference Vector, a novel framework inspired by task arithmetic. Instead of optimizing multiple preferences within a single objective, we train separate models on individual preferences, extract behavior shifts as preference vectors, and dynamically merge them at test time. This modular approach enables fine-grained, user-controllable preference adjustments and facilitates seamless integration of new preferences without retraining. Experiments show that our proposed Preference Vector framework improves helpfulness without excessive conservatism, allows smooth control over preference trade-offs, and supports scalable multi-preference alignment.

Figures (5)

• Figure 1: Overall pipeline. We begin by constructing both positive and negative variants of each preference from the multi-preference dataset. In the first stage, we fine-tune single-preference base models using DPO. In the second stage, we extract Preference Vectors via parameter-wise subtraction between models trained with opposite preferences. In the final stage, we combine these task vectors and apply them to a base model, achieving controllable and extensible multi-preference alignment.
• Figure 2: Preference Vector Scaling with Preference Model Evaluation. We evaluate the controllability of our method on LLaMA3-8B using preference models under varying scaling coefficients $\eta_{Helpful}, \eta_{Harmless} \in \{-1.0, -0.5, 0.0, +0.5, +1.0\}$ for the preference vectors. Green indicates higher helpfulness or harmlessness, while red indicates low ones. The results show relatively smooth and interpretable trends, demonstrating fine-grained control over preference strength.
• Figure 3: Safety, helpfulness, and commonsense performance on different scaling coefficients.The models maintains knowledge base when adding preference vector. ($\eta=\eta_{Helpful}=\eta_{Harmless}$)
• Figure 4: Eigenvalues of different preference vectors obtained from different random seeds. The largest eigenvalue ($\lambda_1$) dominates the others, indicating that preference vectors primarily align along a single, dominant direction.
• Figure 5: layerwise Cosine similarity Averaged for each Component.