Tangent Space Fine-Tuning for Directional Preference Alignment in Large Language Models

TL;DR

This work tackles multi-objective alignment in large language models by reframing preference learning as tangent-space optimization. By freezing a base model and learning per-objective tangent updates, TS-DPO enables linear, composable control over trade-offs (e.g., helpfulness vs. verbosity) at inference without retraining. Empirical results show TS-DPO achieves broader Pareto coverage and smoother control than scalarized DPO, with geometry analyses suggesting clearer disentanglement of preference directions. The approach offers a practical path toward modular, user-controllable AI behaviors, while inviting further scaling and comparisons to related modular alignment methods.

Abstract

Our goal is to enable large language models (LLMs) to balance multiple human preference dimensions; such as helpfulness, safety, and verbosity, through principled and controllable alignment. Existing preference optimization methods, including Direct Preference Optimization (DPO), collapse feedback into a single scalar reward, fixing one balance among objectives and preventing traversal of the Pareto front. Recent work by Ortiz-Jimenez et al. (2023) showed that fine-tuning can be viewed in a model's tangent space, where linearized updates act as additive vectors that can be composed to jointly perform well on multiple tasks. Building on this formulation, we extend this idea to preference alignment and propose Tangent-Space Direct Preference Optimization (TS-DPO), which performs DPO within this locally linear regime to learn per-objective update directions. These directions can be linearly combined at inference to generate user-specified behaviors without additional optimization. Evaluated on the helpfulness-verbosity trade-off using the HelpSteer and UltraFeedback datasets, TS-DPO achieves broader Pareto-optimal coverage and smoother preference control than scalarized DPO. Canonical Correlation Analysis (CCA) further shows that tangent-space training amplifies canonical directions aligned with distinct preferences, improving disentanglement.

Figures (6)

• Figure 1: Pareto frontiers for controllable alignment along the helpfulness–verbosity axes. Each curve corresponds to a different sweep over the mixing coefficients $(\lambda_1,\lambda_2)$, including Convex, Affine, and Affine-2 constructions. The TS-DPO variants produce smoother frontiers, achieving higher helpfulness at comparable verbosity than Task-Vector DPO (DPO). The scalarized DPO baseline (DPO Mixed) is shown for three different points corresponding to various learning rates.
• Figure 2: Layerwise geometry of preference updates for DPO and TS-DPO. We decompose parameter deltas into attention and MLP blocks and measure (a) cosine similarity between helpfulness and verbosity directions and (b) $\ell_2$-norm of helpfulness updates across layers.
• Figure 3: Canonical correlation spectrum between helpfulness and verbosity activation deltas for DPO and TS-DPO.
• Figure 4: TS-DPO tranining losses for various learning rates (a) helpfulness (b) verbosity alignment.
• Figure 5: Task Vector DPO tranining losses for various learning rates for (a) helpfulness (b) verbosity alignment.