Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback

Gihoon Kim, Euntai Kim

Abstract

Reinforcement Learning from Human Feedback (RLHF) is a widely used approach to align large-scale AI systems with human values. However, RLHF typically assumes a single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, we found that VPL suffers from posterior collapse. While this phenomenon is well known in VAEs, it has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables to be ignored, reverting to a single-reward model. To overcome this limitation, we propose Swap-guided Preference Learning (SPL). The key idea is to construct fictitious swap annotators and use the mirroring property of their preferences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, enriches user-specific latents, and improves preference prediction. Our code and data are available at https://github.com/cobang0111/SPL

Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback

Abstract

Reinforcement Learning from Human Feedback (RLHF) is a widely used approach to align large-scale AI systems with human values. However, RLHF typically assumes a single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, we found that VPL suffers from posterior collapse. While this phenomenon is well known in VAEs, it has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables to be ignored, reverting to a single-reward model. To overcome this limitation, we propose Swap-guided Preference Learning (SPL). The key idea is to construct fictitious swap annotators and use the mirroring property of their preferences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, enriches user-specific latents, and improves preference prediction. Our code and data are available at https://github.com/cobang0111/SPL
Paper Structure (37 sections, 4 theorems, 54 equations, 8 figures, 14 tables, 1 algorithm)

This paper contains 37 sections, 4 theorems, 54 equations, 8 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary fundamentals
  3. Reinforcement Learning from Human Feedback
  4. Variational Approach for Personalized Alignment
  5. Inverse Autoregressive Flow
  6. Motivation
  7. Method
  8. Encoding user preferences into a latent
  9. Swap-guided Base Regularization
  10. Preferential Inverse Autoregressive Flow
  11. Decoding personalized rewards from latents
  12. Adaptive Latent Conditioning
  13. Objective function of SPL
  14. Experiments
  15. Baselines
  16. ...and 22 more sections

Key Result

Lemma 1

Let us suppose that ${\bm{z}}_0$ and ${\bm{z}}_{0,\text{swap}}$ are warped to ${\bm{z}}_K$ and ${\bm{z}}_{K,\text{swap}}$ respectively, by P-IAF. Then, the swap probability error $\delta_p$ given in Eq.(eq:swap_prob_error) is bounded by where the reward violation $\delta_{r,K}\triangleq \bigl|\Delta r_\phi({\bm{z}}_K) + \Delta r_\phi(-{\bm{z}}_K)\bigr|$ and the latent mismatch $\delta_{z,K}\trian

Figures (8)

  • Figure 1: Overview of SPL. We propose Swap-guided Preference Learning (SPL), a new framework for personalized alignment. RLHF ouyang2022training cannot adequately reflect user diversity. To overcome this limitation, VPL poddar2024personalizing encodes text data consisting of a prompt $x$ and response $y$ into a single latent embedding. However, this encoding process is highly prone to collapse. In contrast, SPL leverages the structural properties of preference data through guiding mechanisms and a Preferential Inverse Autoregressive flow, allowing the latent space to capture user-specific characteristics.
  • Figure 2: Posterior collapse in Variational Preference Learning. We visualize latent embeddings ${\bm{z}}$ from the VPL encoder using 2D UMAP mcinnes2018umap. Each point denotes a user, colored by their preference type. (a) User preference types are distinctly separated, indicating non-collapse. (b), (c) Latent collapse occurs, making preference types indistinguishable.
  • Figure 3: Differences in posterior distribution between original and swapped inputs. We test how the VPL encoder’s posterior responds when each preference pair is inverted to simulate a user with opposite choices, using the simple dataset Pets. (a) Average RMSE between original and swapped inputs across posterior mean ${\bm{\mu}}$ and log-variance $\boldsymbol{\ell}$. Collapse appears in Llama-3.1-8B (orange), where both parameters remain unchanged, whereas Llama-3.2-3B (green) shows distinct behavior. (b) Plot ${\bm{\mu}}$ vs. ${\bm{\mu}}_\text{swap}$ for Llama-3.2-3B; ${\bm{\mu}} + {\bm{\mu}}_\text{swap}$ is in the lower panel. Initially, the curves are similar, but their difference grows and stabilizes as learning continues, resulting in a sign-reversal.
  • Figure 4: Preference encoding process of SPL
  • Figure 5: Latent embeddings learned on the UF-P dataset. We visualize latent embeddings ${\bm{z}}$ from baselines and SPL (Ours) encoder using 2D t-SNE maaten2008visualizing. Each point denotes a user, colored by their preference type. Compared to the VPL, SPL yields much clearer separation in the latent space.
  • ...and 3 more figures

Theorems & Definitions (8)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3: Transformed mismatch of P-IAF
  • proof
  • Lemma 4: Transformed mismatch of IAF
  • proof