Reinforcement Learning from Meta-Evaluation: Aligning Language Models Without Ground-Truth Labels

TL;DR

RLME offers a scalable route to align language models without ground-truth labels by using natural-language meta-evaluations as rewards. It combines assessment prompting with a GRPO/CISPO-based learning loop to adjust the generator according to evaluator-provided signals. The experiments show RLME can approach label-based RL performance in verifiable tasks, enables multi-objective control, and generalizes to open-domain settings, while revealing vulnerabilities to reward hacking and mitigation strategies. Overall, RLME complements RL-based alignment methods, expanding the domains where such approaches are feasible.

Abstract

Most reinforcement learning (RL) methods for training large language models (LLMs) require ground-truth labels or task-specific verifiers, limiting scalability when correctness is ambiguous or expensive to obtain. We introduce Reinforcement Learning from Meta-Evaluation (RLME), which optimizes a generator using reward derived from an evaluator's answers to natural-language meta-questions (e.g., "Is the answer correct?" or "Is the reasoning logically consistent?"). RLME treats the evaluator's probability of a positive judgment as a reward and updates the generator via group-relative policy optimization, enabling learning without labels. Across a suite of experiments, we show that RLME achieves accuracy and sample efficiency comparable to label-based training, enables controllable trade-offs among multiple objectives, steers models toward reliable reasoning patterns rather than post-hoc rationalization, and generalizes to open-domain settings where ground-truth labels are unavailable, broadening the domains in which LLMs may be trained with RL.

Figures (7)

• Figure 1: Overview of RLME. After generating an answer, one or more evaluators (may be the same model) assign probabilities to natural-language meta-questions about the output. These probabilities are aggregated into a scalar reward, which is then used to update the generative policy via reinforcement learning. This allows models to be tuned even when ground-truth answers are unavailable.
• Figure 2: Comparison of RLME to an RLVR baseline that has access to ground-truth answers. Both methods rapidly exceed $90\%$ accuracy on GSM8K, and RLME closely tracks RLVR despite never observing the true answer.
• Figure 3: RLME performance using different generators with a fixed evaluator (frozen Qwen3-4B-Base). Generator models have a large effect on accuracy.
• Figure 4: RLME performance using different evaluators with a fixed generator (Qwen3-4B-Base). For the Qwen3 evaluator, we compare a live self-evaluator (co-evolving with the generator) to a frozen evaluator (fixed snapshot at initialization). For other evaluators, we only use frozen weights.
• Figure 5: RLME eventually suffers a sharp degradation in accuracy despite continued increases in reward, indicative of reward hacking: the generator learns to exploit weaknesses in the evaluator instead of producing correct solutions. Including a small fraction of prompts with ground-truth answers in the evaluator template (10% for RLME-10GT and 1% for RLME-1GT) stabilizes training and prevents collapse.