Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization

TL;DR

The work addresses the need for efficient, personalized steering of large language models without full fine-tuning. It introduces Bi-directional Preference Optimization (BiPO), which learns steering vectors by directly modulating generation probabilities for target versus opposite behaviors, enabling tunable intensity and preserving base utility. Empirical results across AI personas, truthfulness, hallucination, and jailbreaking show that BiPO outperforms contrastive-activation baselines, transfers across models and LoRA refinements, and yields synergistic effects when combining vectors. The findings highlight practical, resource-efficient pathways for customizable LLM behavior with broad applicability and robustness in alignment-sensitive scenarios.

Abstract

Researchers have been studying approaches to steer the behavior of Large Language Models (LLMs) and build personalized LLMs tailored for various applications. While fine-tuning seems to be a direct solution, it requires substantial computational resources and may significantly affect the utility of the original LLM. Recent endeavors have introduced more lightweight strategies, focusing on extracting "steering vectors" to guide the model's output toward desired behaviors by adjusting activations within specific layers of the LLM's transformer architecture. However, such steering vectors are directly extracted from the activations of human preference data and thus often lead to suboptimal results and occasional failures, especially in alignment-related scenarios. This work proposes an innovative approach that could produce more effective steering vectors through bi-directional preference optimization. Our method is designed to allow steering vectors to directly influence the generation probability of contrastive human preference data pairs, thereby offering a more precise representation of the target behavior. By carefully adjusting the direction and magnitude of the steering vector, we enabled personalized control over the desired behavior across a spectrum of intensities. Extensive experimentation across various open-ended generation tasks, particularly focusing on steering AI personas, has validated the efficacy of our approach. Moreover, we comprehensively investigate critical alignment-concerning scenarios, such as managing truthfulness, mitigating hallucination, and addressing jailbreaking attacks. Remarkably, our method can still demonstrate outstanding steering effectiveness across these scenarios. Furthermore, we showcase the transferability of our steering vectors across different models/LoRAs and highlight the synergistic benefits of applying multiple vectors simultaneously.

Figures (15)

• Figure 1: The comparison results on steering the AI personas of Llama-2-7b-chat-hf model.
• Figure 2: The comparison results on steering truthfulness and hallucination of Llama-2-7b-chat-hf.
• Figure 3: The transferability of our steering vector.
• Figure 4: Sweep different middle layers and perform CAA on Mistral-7B-Instruct-v0.2.
• Figure 5: The comparison results on steering the personas of Mistral-7B-Instruct-v0.2.