Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learning

TL;DR

The paper tackles the challenge of faithfully personalizing large language models by addressing shortcomings of supervised fine-tuning and scalar RLHF rewards, notably reward hacking. It introduces Critique-Post-Edit reinforcement learning, powered by a Generative Reward Model (GRM) that produces multi-dimensional scores plus textual critiques, and an editing-based learning loop where the policy revises its outputs in response to critiques. By training the GRM on rich, paired critique data and combining on-policy originals with off-policy edited samples, the approach yields more targeted and stable learning. Empirically, the method achieves an average 11% win-rate improvement over PPO baselines for Qwen2.5-7B and even surpasses GPT-4.1 for the larger Qwen2.5-14B, demonstrating a practical path to faithful, efficient, and controllable personalization.

Abstract

Faithfully personalizing large language models (LLMs) to align with individual user preferences is a critical but challenging task. While supervised fine-tuning (SFT) quickly reaches a performance plateau, standard reinforcement learning from human feedback (RLHF) also struggles with the nuances of personalization. Scalar-based reward models are prone to reward hacking which leads to verbose and superficially personalized responses. To address these limitations, we propose Critique-Post-Edit, a robust reinforcement learning framework that enables more faithful and controllable personalization. Our framework integrates two key components: (1) a Personalized Generative Reward Model (GRM) that provides multi-dimensional scores and textual critiques to resist reward hacking, and (2) a Critique-Post-Edit mechanism where the policy model revises its own outputs based on these critiques for more targeted and efficient learning. Under a rigorous length-controlled evaluation, our method substantially outperforms standard PPO on personalization benchmarks. Personalized Qwen2.5-7B achieves an average 11\% win-rate improvement, and personalized Qwen2.5-14B model surpasses the performance of GPT-4.1. These results demonstrate a practical path to faithful, efficient, and controllable personalization.

Figures (6)

• Figure 1: Performance curve of SFT, DPO and RL with increasing data size.
• Figure 2: An illustrative “reward hacking” case from RL training with a BT reward model. The model learns to exploit a shortcut by explicitly mention persona traits to get higher reward scores.
• Figure 3: Overview of the Critique-Post-Edit framework. (1) The policy model $\pi_\theta$ generates an original response $y_o$. The GRM evaluates $y_o$ and provides critique, guiding the model to create an edited response $y_e$. (2) The GRM then provides rewards for both $y_o$ and $y_e$, denoted as $R_o$ and $R _e$, forming a candidate pool from which a sampling strategy selects edited samples to combine with the original ones into a training batch for the policy update. Sampling strategies include: Random (selecting a random subset), Reward Rank (selecting the highest-reward samples), and Conditional (selecting if the reward exceeds the original reward).
• Figure 4: Comparison between BT reward model and GRM in PPO: length of rollout (during training) and length-controlled winrate of checkpoints, Lines: Response Length (left axis), Bars: Win Rate (right axis)
• Figure 5: Model performance analysis under different GRM scales