Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist

Kwangwook Seo, Dongha Lee

TL;DR

P-Check introduces a dynamic, checklist-based approach to personalize reward modeling by learning query-specific evaluation criteria from user history. It couples a checklist generator with a contrastive, saliency-driven weighting scheme to produce personalized, discriminative criteria that guide reward prediction. Empirical results show improved reward accuracy and better downstream personalized generation, with robust performance in out-of-distribution and sparse-history scenarios. The framework also demonstrates the checklist's value as informative feedback to policy models, offering a transparent and adaptable path toward on-the-fly personalized alignment.

Abstract

Recent approaches in personalized reward modeling have primarily focused on leveraging user interaction history to align model judgments with individual preferences. However, existing approaches largely treat user context as a static or implicit conditioning signal, failing to capture the dynamic and multi-faceted nature of human judgment. In this paper, we propose P-Check, a novel personalized reward modeling framework, designed to train a plug-and-play checklist generator that synthesizes dynamic evaluation criteria for guiding the reward prediction. To better align these checklists with personalized nuances, we introduce Preference-Contrastive Criterion Weighting, a training strategy that assigns saliency scores to criteria based on their discriminative power for personalized judgment. We conduct extensive experiments and demonstrate that P-Check not only improves reward accuracy but also enhances downstream personalized generation, and remains robust in OOD scenarios.

P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist

TL;DR

P-Check introduces a dynamic, checklist-based approach to personalize reward modeling by learning query-specific evaluation criteria from user history. It couples a checklist generator with a contrastive, saliency-driven weighting scheme to produce personalized, discriminative criteria that guide reward prediction. Empirical results show improved reward accuracy and better downstream personalized generation, with robust performance in out-of-distribution and sparse-history scenarios. The framework also demonstrates the checklist's value as informative feedback to policy models, offering a transparent and adaptable path toward on-the-fly personalized alignment.

Abstract

Recent approaches in personalized reward modeling have primarily focused on leveraging user interaction history to align model judgments with individual preferences. However, existing approaches largely treat user context as a static or implicit conditioning signal, failing to capture the dynamic and multi-faceted nature of human judgment. In this paper, we propose P-Check, a novel personalized reward modeling framework, designed to train a plug-and-play checklist generator that synthesizes dynamic evaluation criteria for guiding the reward prediction. To better align these checklists with personalized nuances, we introduce Preference-Contrastive Criterion Weighting, a training strategy that assigns saliency scores to criteria based on their discriminative power for personalized judgment. We conduct extensive experiments and demonstrate that P-Check not only improves reward accuracy but also enhances downstream personalized generation, and remains robust in OOD scenarios.
Paper Structure (53 sections, 7 equations, 8 figures, 13 tables)

This paper contains 53 sections, 7 equations, 8 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Preliminary Analysis
  3. P-Check: Proposed Method
  4. Problem Formulation
  5. Collecting Checklist from Preference Data
  6. Preference-Contrastive Criterion Weighting
  7. Learning to Synthesize Dynamic Checklist
  8. Personalized Rewarding with Checklist
  9. Experiments
  10. Experimental Setups
  11. Results and Analysis
  12. P-Check helps LLM-as-a-Judge in predicting personalized reward.
  13. P-Check effectively generalizes across OOD benchmarks.
  14. P-Check shows robustness in sparse user interaction scenarios.
  15. P-Check consistently enhances reward accuracy across diverse LLM-Judges.
  16. ...and 38 more sections

Figures (8)

  • Figure 1: Motivating example of P-Check. Unlike the static persona that struggles to capture actionable criteria, dynamic checklist provides reliable guidance by explicitly specifying query-specific requirements.
  • Figure 2: (Upper) Analysis 1. (Lower) Analysis 2.
  • Figure 3: Overview of P-Check, illustrating the training and inference processes.
  • Figure 4: Experiments on sparse interaction scenario.
  • Figure 5: Results on refining the output of policy model with feedback from P-Check's inferred checklist.
  • ...and 3 more figures