One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment

TL;DR

This work tackles the problem of aligning LLMs to individual user preferences under sparse feedback and unseen users. It introduces Meta Reward Modeling (MRM), a meta-learning framework that learns a highly adaptable initialization for per-user reward weights over a shared set of base reward functions, enabling rapid, few-shot personalization. To ensure robustness across diverse user preferences, it adds the Robust Personalization Objective (RPO), which emphasizes hard-to-learn users during meta-optimization. Empirical results on PRISM and Reddit TLDR demonstrate that MRM achieves stronger few-shot personalization and robustness than baselines, while remaining parameter-efficient and scalable for large user populations.

Abstract

Alignment of Large Language Models (LLMs) aims to align outputs with human preferences, and personalized alignment further adapts models to individual users. This relies on personalized reward models that capture user-specific preferences and automatically provide individualized feedback. However, developing these models faces two critical challenges: the scarcity of feedback from individual users and the need for efficient adaptation to unseen users. We argue that addressing these constraints requires a paradigm shift from fitting data to learn user preferences to learn the process of preference adaptation. To realize this, we propose Meta Reward Modeling (MRM), which reformulates personalized reward modeling as a meta-learning problem. Specifically, we represent each user's reward model as a weighted combination of base reward functions, and optimize the initialization of these weights using a Model-Agnostic Meta-Learning (MAML)-style framework to support fast adaptation under limited feedback. To ensure robustness, we introduce the Robust Personalization Objective (RPO), which places greater emphasis on hard-to-learn users during meta optimization. Extensive experiments on personalized preference datasets validate that MRM enhances few-shot personalization, improves user robustness, and consistently outperforms baselines.

Figures (7)

• Figure 1: Comparison of personalized reward modeling methods: (a) Personalized input incorporates user contexts; (b) Personalized parameter assigns user-specific parameters; (c) Meta Reward Modeling formulates personalization as a meta-learning problem by learning an adaptable initialization.
• Figure 2: Overview of Meta Reward Modeling. The model employs base reward functions with shared weight initialization, adapts user-specific weights in the inner loop, and updates both initialization and base functions in the outer loop with the robust personalization objective.
• Figure 3: Performance of average accuracy on the worst 10%, 20%, and 50% of users for (a) PRISM and (b) Reddit TLDR with 100 examples. MRM consistently outperforms baselines on all proportions of worst users, showing stronger robustness.
• Figure 4: Effect of threshold ratio on PRISM. (a) Overall accuracy with different threshold ratios ($\rho=0.1, 0.2, 0.5$). (b) Accuracy on the worst $k\%$ of users ($k=10,20,50$).
• Figure 5: Performance with respect to (a) meta batch size and (b) smoothing parameter $\gamma$. In (b), $\gamma{=}0^*$ denotes hard filtering.