Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model

Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, Ying-Cong Chen

TL;DR

This work introduces PARM, a single, preference-conditioned Autoregressive Reward Model for multi-objective test-time alignment of frozen LLMs. It proposes PBLoRA to condition the ARM on a low-dimensional preference vector, enabling explicit trade-off control across objectives and sharing learning across dimensions to cover the Pareto front. Training is done via scalarized objectives over sampled preference vectors, yielding a model that can guide generation for any given preference without retraining. Experiments on safety alignment and helpful-assistant tasks demonstrate improved alignment (HV and MIP) and a weak-to-strong capability, where a small PARM guides a much larger frozen model efficiently.

Abstract

Multi-objective test-time alignment aims to adapt large language models (LLMs) to diverse multi-dimensional user preferences during inference while keeping LLMs frozen. Recently, GenARM (Xu et al., 2025) first independently trains Autoregressive Reward Models (ARMs) for each preference dimension without awareness of each other, then combines their outputs based on user-specific preference vectors during inference to achieve multi-objective test-time alignment, leading to two key limitations: the need for \textit{multiple} ARMs increases the inference cost, and the separate training of ARMs causes the misalignment between the guided generation and the user preferences. To address these issues, we propose Preference-aware ARM (PARM), a single unified ARM trained across all preference dimensions. PARM uses our proposed Preference-Aware Bilinear Low-Rank Adaptation (PBLoRA), which employs a bilinear form to condition the ARM on preference vectors, enabling it to achieve precise control over preference trade-offs during inference. Experiments demonstrate that PARM reduces inference costs and achieves better alignment with preference vectors compared with existing methods. Additionally, PARM enables weak-to-strong guidance, allowing a smaller PARM to guide a larger frozen LLM without expensive training, making multi-objective alignment accessible with limited computing resources. The code is available at https://github.com/Baijiong-Lin/PARM.

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model

TL;DR

This work introduces PARM, a single, preference-conditioned Autoregressive Reward Model for multi-objective test-time alignment of frozen LLMs. It proposes PBLoRA to condition the ARM on a low-dimensional preference vector, enabling explicit trade-off control across objectives and sharing learning across dimensions to cover the Pareto front. Training is done via scalarized objectives over sampled preference vectors, yielding a model that can guide generation for any given preference without retraining. Experiments on safety alignment and helpful-assistant tasks demonstrate improved alignment (HV and MIP) and a weak-to-strong capability, where a small PARM guides a much larger frozen model efficiently.

Abstract

Multi-objective test-time alignment aims to adapt large language models (LLMs) to diverse multi-dimensional user preferences during inference while keeping LLMs frozen. Recently, GenARM (Xu et al., 2025) first independently trains Autoregressive Reward Models (ARMs) for each preference dimension without awareness of each other, then combines their outputs based on user-specific preference vectors during inference to achieve multi-objective test-time alignment, leading to two key limitations: the need for \textit{multiple} ARMs increases the inference cost, and the separate training of ARMs causes the misalignment between the guided generation and the user preferences. To address these issues, we propose Preference-aware ARM (PARM), a single unified ARM trained across all preference dimensions. PARM uses our proposed Preference-Aware Bilinear Low-Rank Adaptation (PBLoRA), which employs a bilinear form to condition the ARM on preference vectors, enabling it to achieve precise control over preference trade-offs during inference. Experiments demonstrate that PARM reduces inference costs and achieves better alignment with preference vectors compared with existing methods. Additionally, PARM enables weak-to-strong guidance, allowing a smaller PARM to guide a larger frozen LLM without expensive training, making multi-objective alignment accessible with limited computing resources. The code is available at https://github.com/Baijiong-Lin/PARM.
Paper Structure (21 sections, 1 theorem, 16 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 21 sections, 1 theorem, 16 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary on ARM
  4. PARM: Preference-Aware ARM
  5. Motivations and Problem Formulation
  6. Preference-Aware Bilinear Low-Rank Adaptation
  7. Training of PARM
  8. Guided Generation via PARM
  9. Experiments
  10. Safety Alignment
  11. Helpful Assistant
  12. Ablation Study
  13. Conclusion
  14. Additional Related Work
  15. Multi-Objective Optimization.
  16. ...and 6 more sections

Key Result

Theorem 4.1

Assume both ${\mathbf{B}}$ and ${\mathbf{A}}$ have rank $r$. The outer product of $\{{\mathbf{b}}_i \!:\! i\!=\!1,\cdots,r\}$ with $\{{\mathbf{a}}_i\!:\! i\!=\!1,\cdots,r\}$ results in $r^2$linearly independent matrices $\{\mathbf{b}_i \mathbf{a}_j^\top\!\!\!:\!i\!=\!1,\cdots,r, j\!=\!1,\cdots,r\}$

Figures (4)

  • Figure 1: Learned Pareto fronts of RSrame2023rewarded, MODshi2024decoding, GenARMxu2024genarm, and PARM on the safety alignment task. PARM and GenARM are fine-tuned from the Alpaca-7B model and subsequently used to guide the generation of the frozen Alpaca-7B model.
  • Figure 2: Learned Pareto fronts of MOD-w2sshi2024decoding, GenARMxu2024genarm, and PARM on the safety alignment task. All methods are fine-tuned from the Alpaca-7B model and subsequently used to guide the generation of the frozen Alpaca-65B model.
  • Figure 3: Learned Pareto fronts of MOD-w2sshi2024decoding, GenARMxu2024genarm, and PARM on the helpful assistant task. Figure (a) presents a 3D visualization while Figures (b), (c), and (d) display 2D projections by fixing one of the preference weights to zero. All methods are trained on the TinyLLaMA-1.1B-Chat model and then used to guide the frozen LLaMA-2-7B-Chat's generation.
  • Figure 4: Learned Pareto fronts of different configurations of PBLoRA on the safety alignment task.

Theorems & Definitions (6)

  • Theorem 4.1
  • Definition 2.1: Pareto dominance
  • Definition 2.2: Pareto optimality
  • Definition 2.3: Pareto set
  • Definition 2.4: Pareto front
  • proof