Generalist Reward Models: Found Inside Large Language Models
Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, Zhi-Hua Zhou
TL;DR
The paper shows that a powerful, generalist reward model is already latent inside any LLM trained with next-token prediction, and that this endogenous reward can be recovered without training by interpreting the model’s logits as a soft Q-function via offline IRL. It proves that using this endogenous reward for RLFT yields a provably better error bound than continuing with imitation learning, addressing compounding errors that plague traditional IL approaches. Empirically, the authors demonstrate that EndoRM can outperform training-free reward baselines and even explicit reward models, and that RL with EndoRM yields self-improvement on math-reasoning benchmarks. The work suggests a paradigm shift where reward modeling is replaced by principled elicitation of the model’s own evaluative knowledge, with implications for personalization, distillation, and multi-modal alignment, while acknowledging potential biases and the need for careful prompting and hybrid safeguards.
Abstract
The alignment of Large Language Models (LLMs) is critically dependent on reward models trained on costly human preference data. While recent work explores bypassing this cost with AI feedback, these methods often lack a rigorous theoretical foundation. In this paper, we discover that a powerful generalist reward model is already latently present within any LLM trained via standard next-token prediction. We prove that this endogenous reward is not a heuristic, but is theoretically equivalent to a reward function learned through offline inverse reinforcement learning. This connection allows us to directly elicit a high-quality reward signal from a base (pre-trained or supervised fine-tuned) model without any further training. Critically, we also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model. To our best knowledge, this is the first theoretical proof of the effectiveness of reinforcement learning for LLMs. Our experiments validate this theory, demonstrating that our method not only outperforms existing LLM-as-a-judge approaches but can also surpass explicitly trained reward models. These findings suggest that the reward modeling stage can be replaced by a principled method of eliciting the knowledge already captured during pre-training, heralding a more efficient, powerful, and scalable paradigm for LLMs alignment as well as multi-modal models.