Reward-free Alignment for Conflicting Objectives

TL;DR

The paper tackles multi-objective alignment for large language models under conflicting objectives (e.g., helpfulness vs. harmlessness) using a reward-free approach. It introduces Reward-free Alignment for Conflicted Objectives (RACO), which applies a clipped conflict-averse gradient descent (CAGrad-Clip) to direct preference losses without explicit reward modeling, and proves convergence to weighted Pareto-critical points with acceleration in the two-objective case. The method integrates naturally with DPO-style objectives and remains scalable to high-dimensional LLM fine-tuning. Empirical results on multi-objective summarization and safety alignment across multiple model families show that RACO achieves superior Pareto-frontier trade-offs compared with existing reward-free baselines. The work offers a principled, practical tool for balancing multiple desirable properties in LLMs, enabling more controllable and safer deployments without resorting to reward-modeling pipelines.

Abstract

Direct alignment methods are increasingly used to align large language models (LLMs) with human preferences. However, many real-world alignment problems involve multiple conflicting objectives, where naive aggregation of preferences can lead to unstable training and poor trade-offs. In particular, weighted loss methods may fail to identify update directions that simultaneously improve all objectives, and existing multi-objective approaches often rely on explicit reward models, introducing additional complexity and distorting user-specified preferences. The contributions of this paper are two-fold. First, we propose a Reward-free Alignment framework for Conflicted Objectives (RACO) that directly leverages pairwise preference data and resolves gradient conflicts via a novel clipped variant of conflict-averse gradient descent. We provide convergence guarantees to Pareto-critical points that respect user-specified objective weights, and further show that clipping can strictly improve convergence rate in the two-objective setting. Second, we improve our method using some heuristics and conduct experiments to demonstrate the compatibility of the proposed framework for LLM alignment. Both qualitative and quantitative evaluations on multi-objective summarization and safety alignment tasks across multiple LLM families (Qwen 3, Llama 3, Gemma 3) show that our method consistently achieves better Pareto trade-offs compared to existing multi-objective alignment baselines.

Figures (6)

• Figure 1: Basic weighted-sum loss combines gradients $w_1 g_1$ and $w_2 g_2$, which can miss a direction that improves both objectives (left); Correction gradient $G_0$ in CAGrad may over-correct toward the less-preferred objective (middle); CAGrad-Clip limits the correction using the preference weights (brown dashed line), yielding an update that better respects the intended trade-off (right).
• Figure 2: (a) Comparison of training-time margin dynamics for RACO, AMoPO, and DPO LW on Qwen3-4B, with validation takes every 100 steps; (b) Additional results of margin comparison over Llama3-8B training; (c) Final Pareto frontiers of trade-off between summary conciseness and generation quality for Qwen3-4B and Llama3-8B, across input weights $w_{\text{qual}} \in \{0.8, 0.65, 0.5, 0.35, 0.2\}$.
• Figure 3: Pareto frontiers between summary faithfulness and quality across different input weights. "PA" indicates the pre-alignment model performance before alignment training.
• Figure 4: Pareto frontiers under input weights $\{0.2, 0.35, 0.5, 0.65, 0.8\}$ illustrating the trade-off between response harmlessness and helpfulness for Qwen3 and Gemma3 base models (with SFT) and instruction-finetuned models. Higher values indicate better performance on both harmlessness and helpfulness. "PA" denotes the pre-alignment model performance before training.
• Figure 5: Qualitative ablation results for $p_i$-clipping with objective input weights $\{w_{1}, w_{2}\}=\{0.8, 0.2\}$. Columns 1 and 2 show the correction weight $p_i$ for objective $i$, with and without clipping. Columns 3 and 4 report the validation objective margin $m_i$ for objective $i$.

Theorems & Definitions (17)

• Definition 2.1
• Definition 2.2
• Theorem 3.1: Convergence
• Theorem 3.2: Acceleration
• Lemma 2.1
• proof
• Lemma 2.2
• proof
• Lemma 2.3
• proof