Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAD: Personalized Alignment of LLMs at Decoding-Time

Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, Zuozhu Liu

TL;DR

This work tackles the challenge of aligning LLM outputs to diverse personalized preferences without costly retraining. It introduces Personalized Alignment at Decoding-time (PAD), which decouples generation dynamics from personalized preferences via a token-level MDP and a single PersRM, and uses a guided-decoding rule that weights base-model likelihoods by a personalized reward signal. The approach yields superior alignment, including to unseen preferences, and demonstrates model-agnostic scalability across multiple base models while incurring moderate decoding-time overhead. PAD thus enables real-time, personalized LLM alignment suitable for broad user bases without additional policy training.

Abstract

Aligning with personalized preferences, which vary significantly across cultural, educational, and political differences, poses a significant challenge due to the computational costs and data demands of traditional alignment methods. In response, this paper presents Personalized Alignment at Decoding-time (PAD), a novel framework designed to align LLM outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training. By introducing a unique personalized reward modeling strategy, this framework decouples the text generation process from personalized preferences, facilitating the generation of generalizable token-level personalized rewards. The PAD algorithm leverages these rewards to guide the decoding process, dynamically tailoring the base model's predictions to personalized preferences. Extensive experimental results demonstrate that PAD not only outperforms existing training-based alignment methods in terms of aligning with diverse preferences but also shows significant generalizability to preferences unseen during training and scalability across different base models. This work advances the capability of LLMs to meet user needs in real-time applications, presenting a substantial step forward in personalized LLM alignment.

PAD: Personalized Alignment of LLMs at Decoding-Time

TL;DR

This work tackles the challenge of aligning LLM outputs to diverse personalized preferences without costly retraining. It introduces Personalized Alignment at Decoding-time (PAD), which decouples generation dynamics from personalized preferences via a token-level MDP and a single PersRM, and uses a guided-decoding rule that weights base-model likelihoods by a personalized reward signal. The approach yields superior alignment, including to unseen preferences, and demonstrates model-agnostic scalability across multiple base models while incurring moderate decoding-time overhead. PAD thus enables real-time, personalized LLM alignment suitable for broad user bases without additional policy training.

Abstract

Aligning with personalized preferences, which vary significantly across cultural, educational, and political differences, poses a significant challenge due to the computational costs and data demands of traditional alignment methods. In response, this paper presents Personalized Alignment at Decoding-time (PAD), a novel framework designed to align LLM outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training. By introducing a unique personalized reward modeling strategy, this framework decouples the text generation process from personalized preferences, facilitating the generation of generalizable token-level personalized rewards. The PAD algorithm leverages these rewards to guide the decoding process, dynamically tailoring the base model's predictions to personalized preferences. Extensive experimental results demonstrate that PAD not only outperforms existing training-based alignment methods in terms of aligning with diverse preferences but also shows significant generalizability to preferences unseen during training and scalability across different base models. This work advances the capability of LLMs to meet user needs in real-time applications, presenting a substantial step forward in personalized LLM alignment.
Paper Structure (56 sections, 21 equations, 6 figures, 5 tables, 2 algorithms)

This paper contains 56 sections, 21 equations, 6 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Large language model alignment.
  4. Personalized alignment.
  5. Method
  6. Preliminaries
  7. Text generation as token-level markov decision process.
  8. The RLHF Pipeline.
  9. Personalized alignment within MDP
  10. Personalized alignment at Decoding-time
  11. Generalized personalized alignment
  12. Practical Implementations
  13. Optimization
  14. Guided Decoding
  15. Experiment
  16. ...and 41 more sections

Figures (6)

  • Figure 1: An illustration of the inference phase for personalized alignment at decoding-time (PAD) with optimized personalized reward model (PersRM). Given the personalized preference and the current context, we first calculate the probability distribution of the base model for the next token. Then, we calculate the reward from PersRM combining features of current state and personalized weight. Finally, the next token can be selected based on the weighted scores.
  • Figure 2: Alignment results for pre-defined preferences related to harmless, helpful, and humor.
  • Figure 3: Alignment on Customized Preferences. Abbreviations include "Exp" for Expert, "Inf" for Informative, "Cre" for Creative, "Pre" for Preliminary, "Con" for Concise, and "Fac" for Factual.
  • Figure 4: Comparison of various decoding strategies that can be integrated with our PAD.
  • Figure 5: Sensitivity analysis on $\beta$.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Definition 1
  • Definition 2