No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
Yanzhi Zhang, Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, Jiyan He
TL;DR
This paper investigates reinforcement learning from internal feedback (RLIF) as an unsupervised alternative to RLHF and RLVR for improving LLM reasoning. It analyzes three internal reward signals—self-certainty, token-level entropy, and trajectory-level entropy—showing their partial equivalence through a theoretical lens anchored in policy-entropy optimization. Empirically, RLIF provides early gains on base (non-instruction-tuned) models on math benchmarks but suffers performance degradation as training continues, with limited benefits for instruction-tuned models. The authors argue this pattern arises from entropy dynamics: RLIF often reduces policy entropy too quickly, suppressing necessary exploratory reasoning, especially in models with lower initial entropy, and they offer guidelines for applying internal feedback signals to balance exploration and exploitation in post-training settings.
Abstract
Reinforcement learning has emerged as a powerful paradigm for post-training large language models (LLMs) to improve reasoning. Approaches like Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) have shown strong results, but they require extensive external supervision. We investigate an alternative class of methods, Reinforcement Learning from Internal Feedback (RLIF), which relies solely on intrinsic model-derived signals instead of external rewards. In particular, we leverage unsupervised reward proxies such as token-level entropy, trajectory-level entropy, and self-certainty. Our theoretical analysis shows these internal objectives are partially equivalent, and we empirically evaluate various RLIF strategies on challenging math reasoning benchmarks. Experimental results demonstrate that RLIF can boost the reasoning performance of base LLMs at the beginning phase of the training, matching or surpassing RLVR techniques on these tasks. However, when training progresses, performance degrades even below the model before training. Moreover, we find that RLIF yields little improvement for instruction-tuned models, indicating diminishing returns of intrinsic feedback once an LLM is already instruction-tuned. We further analyze this limitation by mixing model weights and explain the reason of RLIF's training behaviors, providing practical guidelines for integrating internal feedback signals into LLM training. We hope our analysis of internal feedback will inform more principled and effective strategies for LLM post-training.