Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models

TL;DR

Reward-based LLM alignment is vulnerable to reward hacking and burdensome prompt engineering. Meta Policy Optimization (MPO) introduces a Meta Reward Model that continuously refines the reward rubric, enabling adaptive, context-aware evaluation signals guided by evaluative thinking and metacognition. Across essay writing, summarization, ethical reasoning, and mathematical reasoning, MPO yields strong alignment improvements, mitigates exploitative behaviors, and often surpasses hand-engineered prompts with fewer manual design efforts. The framework is modular and extensible to other RL methods, offering a scalable path toward more robust and adaptable AI alignment in practice.

Abstract

Reward-based alignment methods for large language models (LLMs) face two key limitations: vulnerability to reward hacking, where models exploit flaws in the reward signal; and reliance on brittle, labor-intensive prompt engineering when LLMs are used as reward models. We introduce Meta Policy Optimization (MPO), a framework that addresses these challenges by integrating a meta-reward model that dynamically refines the reward model's prompt throughout training. In MPO, the meta-reward model monitors the evolving training context and continuously adjusts the reward model's prompt to maintain high alignment, providing an adaptive reward signal that resists exploitation by the policy. This meta-learning approach promotes a more stable policy optimization, and greatly reduces the need for manual reward prompt design. It yields performance on par with or better than models guided by extensively hand-crafted reward prompts. Furthermore, we show that MPO maintains its effectiveness across diverse tasks, from essay writing to mathematical reasoning, without requiring specialized reward designs. Beyond standard RLAIF, MPO's meta-learning formulation is readily extensible to higher-level alignment frameworks. Overall, this method addresses theoretical and practical challenges in reward-based RL alignment for LLMs, paving the way for more robust and adaptable alignment strategies. The code and data can be accessed at: https://github.com/minnesotanlp/mpo

Figures (9)

• Figure 1: In standard RLAIF, the reward model used during proximal policy optimization (PPO) remains fixed throughout RL alignment. In contrast, MPO framework (in green) introduces a meta reward model that dynamically evolves the reward model based on the current training context, including the task prompt, sampled generations with associated scores, and the latest evaluation prompt. MPO leverages this contextual information to iteratively refine the evaluation prompt, enabling more adaptive and effective alignment.
• Figure 2: Dimensions of Evaluative Thinking: Depth and Breadth.
• Figure 3: The three Meta Policy Optimization steps---meta-analysis, meta-refinement, and meta-merging---are carried out by the meta reward model and operate over a broader input context than that used by the reward model.
• Figure 4: Training curves for eight essay‑writing policy models, each pairing different‑sized reward models (RM) and meta‑reward models (MRM). The RL Reward and Normalized RL Reward plots show how reward values evolve over global batch steps, capturing the quality of generated responses as judged by the corresponding RM at each point in training. The normalized plot is obtained by dividing the RL reward values by the total attainable score defined by the current rubric, providing a more consistent view of reward dynamics across evolving evaluation criteria. Kullback-Leibler (KL) divergence quantifies the extent of policy drift throughout training. The dotted vertical lines indicate MPO rounds, which occur every batch size $\times$ MPO step---640 steps in our setup.
• Figure 5: (a) Mean length of rubric items for essay writing task across the MPO-aligned models. (b) Mean normalized total rubric score for 1,000 test essays (generated by the 32b_72b model) across successive evaluation prompt refinements.

Theorems & Definitions (3)

• Remark 1: Depth and Breadth of ET
• Remark 2
• Remark 3