YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation

TL;DR

YaPO introduces learnable sparse activation steering vectors by projecting activations into a pretrained Sparse Autoencoder space and optimizing sparse codes under a BiPO-style preference objective. This yields disentangled, interpretable, and efficient steering directions that improve convergence, stability, and fine-grained cultural alignment, while preserving general knowledge. The authors curate a multilingual cultural alignment benchmark spanning five languages and fifteen contexts, demonstrating YaPO's strong performance across localized and implicit prompts, as well as its generalization to other alignment dimensions like hallucination and jailbreak. The work suggests a general recipe for scalable, robust domain adaptation in LLMs with public code, and shows that sparse, preference-driven steering can outperform dense baselines in both stability and interpretability.

Abstract

Steering Large Language Models (LLMs) through activation interventions has emerged as a lightweight alternative to fine-tuning for alignment and personalization. Recent work on Bi-directional Preference Optimization (BiPO) shows that dense steering vectors can be learned directly from preference data in a Direct Preference Optimization (DPO) fashion, enabling control over truthfulness, hallucinations, and safety behaviors. However, dense steering vectors often entangle multiple latent factors due to neuron multi-semanticity, limiting their effectiveness and stability in fine-grained settings such as cultural alignment, where closely related values and behaviors (e.g., among Middle Eastern cultures) must be distinguished. In this paper, we propose Yet another Policy Optimization (YaPO), a \textit{reference-free} method that learns \textit{sparse steering vectors} in the latent space of a Sparse Autoencoder (SAE). By optimizing sparse codes, YaPO produces disentangled, interpretable, and efficient steering directions. Empirically, we show that YaPO converges faster, achieves stronger performance, and exhibits improved training stability compared to dense steering baselines. Beyond cultural alignment, YaPO generalizes to a range of alignment-related behaviors, including hallucination, wealth-seeking, jailbreak, and power-seeking. Importantly, YaPO preserves general knowledge, with no measurable degradation on MMLU. Overall, our results show that YaPO provides a general recipe for efficient, stable, and fine-grained alignment of LLMs, with broad applications to controllability and domain adaptation. The associated code and data are publicly available\footnote{https://github.com/MBZUAI-Paris/YaPO}.

Figures (5)

• Figure 1: Overview of YaPO. Unlike dense BiPO, which learns entangled steering directions directly in activation space, YaPO leverages a pretrained Sparse Autoencoder (SAE) to project activations into an interpretable sparse space. By optimizing sparse codes, YaPO learns disentangled and robust steering vectors that improve convergence, stability, and cultural alignment, while preserving generalization across domains.
• Figure 2: Localized (a) and non-localized (b) training and evaluation loss comparison between BiPO and YaPO for Egypt (a) and Nepal (b).
• Figure 3: Training accuracy over epochs for YaPO (red), BiPO (blue), and the unsteered baseline (orange) on the MCQ localization task across six cultural regions.
• Figure 4: Effect of steering multiplier $\lambda$ on MCQ accuracy across methods for different cultural settings. YaPO exhibits smoother and more stable accuracy scaling compared to dense baselines.
• Figure 5: Activation patching analysis on Gemma-2-2B. We intervene across layers to trace cultural features in model representations. The plots show the probability of producing culturally specific answers (Egypt, Morocco) versus Western defaults as activations are patched. We empirically identify layer 15 as the most culturally relevant layer.

Theorems & Definitions (2)

• Definition 1: Performance--Normalized Localization Gap (PNLG)
• Definition 2: Robust Cultural Accuracy (RCA)