Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Users as Annotators: LLM Preference Learning from Comparison Mode

Zhongze Cai, Xiaocheng Li

TL;DR

The paper addresses noisy user-generated preference data collected via LLM comparison-mode by introducing a probabilistic model that separates user attentiveness from population-level preferences. An EM algorithm estimates per-user attentiveness $\eta_j$ and the population mean $\mu$, enabling principled filtering of low-quality annotations. Through synthetic experiments and DPO-based alignment tasks, filtering attentive users consistently improves downstream performance, even when data volume drops. This approach promises scalable, user-driven alignment with built-in data quality control and potential incentive mechanisms to encourage high-quality feedback.

Abstract

Pairwise preference data have played an important role in the alignment of large language models (LLMs). Each sample of such data consists of a prompt, two different responses to the prompt, and a binary label indicating which of the two responses is better. The labels are usually annotated by professional human annotators. In this paper, we consider an alternative approach to collect pairwise preference data -- user annotation from comparison mode. With the increasingly wider adoption of LLMs among the population, users are contributing more and more of their preference labels through their daily interactions with the LLMs. The upside of such labels is that users are the best experts in judging the responses to their own queries/prompts, but the downside is the lack of quality control in these labels. In this paper, we consider a new idea of generating two responses from two different models or two different versions of the same model. The asymmetry allows us to make an inference of the user's data quality through our proposed user behavior model. We develop an expectation-maximization algorithm to estimate a latent quality factor of the user, and filter users' annotation data accordingly. The downstream task shows the effectiveness of our approach in both capturing the user behavior and data filtering for LLM alignment.

Users as Annotators: LLM Preference Learning from Comparison Mode

TL;DR

The paper addresses noisy user-generated preference data collected via LLM comparison-mode by introducing a probabilistic model that separates user attentiveness from population-level preferences. An EM algorithm estimates per-user attentiveness and the population mean , enabling principled filtering of low-quality annotations. Through synthetic experiments and DPO-based alignment tasks, filtering attentive users consistently improves downstream performance, even when data volume drops. This approach promises scalable, user-driven alignment with built-in data quality control and potential incentive mechanisms to encourage high-quality feedback.

Abstract

Pairwise preference data have played an important role in the alignment of large language models (LLMs). Each sample of such data consists of a prompt, two different responses to the prompt, and a binary label indicating which of the two responses is better. The labels are usually annotated by professional human annotators. In this paper, we consider an alternative approach to collect pairwise preference data -- user annotation from comparison mode. With the increasingly wider adoption of LLMs among the population, users are contributing more and more of their preference labels through their daily interactions with the LLMs. The upside of such labels is that users are the best experts in judging the responses to their own queries/prompts, but the downside is the lack of quality control in these labels. In this paper, we consider a new idea of generating two responses from two different models or two different versions of the same model. The asymmetry allows us to make an inference of the user's data quality through our proposed user behavior model. We develop an expectation-maximization algorithm to estimate a latent quality factor of the user, and filter users' annotation data accordingly. The downstream task shows the effectiveness of our approach in both capturing the user behavior and data filtering for LLM alignment.
Paper Structure (40 sections, 10 theorems, 88 equations, 8 figures, 8 tables, 1 algorithm)

This paper contains 40 sections, 10 theorems, 88 equations, 8 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Reward Modeling and User Behavioral Model
  3. Reward Modeling
  4. User Behavioral Model
  5. Attentiveness Inference via Expectation Maximization
  6. Parameter Estimation with EM
  7. Global Convergence Guarantee
  8. Inference of User Attentiveness
  9. Experiments
  10. Synthetic EM Experiments
  11. Data Filtering and LLM Alignment
  12. Discussions and Conclusions
  13. Related Work
  14. Aligning LLMs with Human Preferences
  15. Quality Control in Preference Annotation
  16. ...and 25 more sections

Key Result

Lemma 1

Suppose user $j$'s prompt distribution follows the population level prompt distribution $P_{\mathrm{prompt}}$, the following holds under the user behavioral model (eqn:p_user)

Figures (8)

  • Figure 1: ChatGPT’s comparison mode. Two responses are generated, and the user clicks the preferred one.
  • Figure 2: The proposed pipeline estimates the quality of users’ preference labels in the comparison mode. For each prompt, two responses are generated from different LLMs. Each user’s preference history is tracked over time. When Model A is more powerful than Model B, attentive users are expected to favor Model A more often, whereas casual users might to choose between the two models with equal probability. These preference patterns are then used to fit a user behavior model and infer the attentiveness level of each user. Then only the labels from the attentive users will be used for the downstream alignment tasks.
  • Figure 3: The iterative updates of Algorithm \ref{['alg:EM_UBM']} under the model settings of Example 1 (left) and Example 2 (right). The true model for Example 1 is $\mathcal{P}_\eta = 0.6\cdot \delta_{0.4} + 0.4\cdot \delta_{0.98}$ and for Example 2 is $\mathcal{P}_\eta = \text{Beta}(3,5)$. The x- and y-axes represent the estimated parameter values, with the red star indicating the true parameters. The algorithm is repeated for multiple trials, each starting from different initialization points (marked by circles), and the trajectories of parameter updates (marked by directed arrows) are visualized. Despite different initializations, the trajectories always lead to the true parameters.
  • Figure 4: The estimation of $\mathcal{P}_\eta$ under model misspecification. In the left panel, the "True Distribution" $\mathcal{P}_\eta$ is a mixture of two Beta distributions, while the "Estimation" is taken from the two-point discrete family. In the right panel, the "True Distribution" follows a logistic-normal distribution, while the "Estimation" is taken from the Beta family. In both cases, Algorithm \ref{['alg:EM_UBM']} approximates $\mathcal{P}_\eta$ reasonably well: in the left panel, it captures the bimodality of the true distribution, and in the right, it provides a close approximation to the true density.
  • Figure 5: The histogram of reward scores for different models, evaluated on the UltraFeedback dataset. The average score for each model is shown in brackets.
  • ...and 3 more figures

Theorems & Definitions (19)

  • Lemma 1
  • Theorem 1: Informal
  • Lemma 2
  • proof
  • Definition 1: Glivenko--Cantelli
  • Lemma 3: Lipschitz Criterion, Thm. 2.4.1 of vanderVaart1996
  • Definition 2: Covering Number and Metric Entropy
  • Definition 3: Big-O In Probability
  • Lemma 4: Thm. 2.14.1 of vanderVaart1996
  • Remark 1
  • ...and 9 more