Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution

Kaiyuan Li, Jing-Cheng Pang, Yang Yu

Abstract

Reinforcement learning from verifiable rewards (RLVR) stimulates the thinking processes of large language models (LLMs), substantially enhancing their reasoning abilities on verifiable tasks. It is often assumed that similar gains should transfer to general question answering (GQA), but this assumption has not been thoroughly validated. To assess whether RLVR automatically improves LLM performance on GQA, we propose a Cross-Generation evaluation framework that measures the quality of intermediate reasoning by feeding the generated thinking context into LLMs of varying capabilities. Our evaluation leads to a discouraging finding: the efficacy of the thinking process on GQA tasks is markedly lower than on verifiable tasks, suggesting that explicit training on GQA remains necessary in addition to training on verifiable tasks. We further observe that direct RL training on GQA is less effective than RLVR. Our hypothesis is that, whereas verifiable tasks demand robust logical chains to obtain high rewards, GQA tasks often admit shortcuts to high rewards without cultivating high-quality thinking. To avoid possible shortcuts, we introduce a simple method, Separated Thinking And Response Training (START), which first trains only the thinking process, using rewards defined on the final answer. We show that START improves both the quality of thinking and the final answer across several GQA benchmarks and RL algorithms.

RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution

Abstract

Reinforcement learning from verifiable rewards (RLVR) stimulates the thinking processes of large language models (LLMs), substantially enhancing their reasoning abilities on verifiable tasks. It is often assumed that similar gains should transfer to general question answering (GQA), but this assumption has not been thoroughly validated. To assess whether RLVR automatically improves LLM performance on GQA, we propose a Cross-Generation evaluation framework that measures the quality of intermediate reasoning by feeding the generated thinking context into LLMs of varying capabilities. Our evaluation leads to a discouraging finding: the efficacy of the thinking process on GQA tasks is markedly lower than on verifiable tasks, suggesting that explicit training on GQA remains necessary in addition to training on verifiable tasks. We further observe that direct RL training on GQA is less effective than RLVR. Our hypothesis is that, whereas verifiable tasks demand robust logical chains to obtain high rewards, GQA tasks often admit shortcuts to high rewards without cultivating high-quality thinking. To avoid possible shortcuts, we introduce a simple method, Separated Thinking And Response Training (START), which first trains only the thinking process, using rewards defined on the final answer. We show that START improves both the quality of thinking and the final answer across several GQA benchmarks and RL algorithms.
Paper Structure (37 sections, 4 equations, 5 figures, 9 tables)

This paper contains 37 sections, 4 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The Thinking Stagnation of RL on GQA
  3. Uncovering the Limited Efficacy of Thinking in GQA
  4. The Stagnation of Thinking in Reinforcement Learning on GQA
  5. Separated Thinking And Response Training (START)
  6. Preliminaries: RL for LLMs as an MDP
  7. Conceptual Shift: From Exploration Space to Environment
  8. Phase I: Thinking Evolution via Gradient Masking
  9. Phase II: Joint Optimization
  10. Integrated with Standard RL Algorithms
  11. Experiment
  12. Setup
  13. Main Result
  14. Analysis of Phase I: Thinking as a Foundation
  15. Exploring the Emergence of Meta-Context Modeling
  16. ...and 22 more sections

Figures (5)

  • Figure 1: Cross-generation performance heatmaps Comparing the influence of thinking vs. answering model capacity. (Left) In reasoning tasks, performance is almost entirely dictated by the quality of the thinking trace. (Right) In general tasks, the answering model capacity remains a dominant factor, and the performance provided by superior thinking is less decisive.
  • Figure 2: Overall framework of START method.
  • Figure 3: Reward curves during reinforcement learning fine-tuning. We compare the reward curves of GRPO, GRPO-MA, and our proposed GRPO+START on the ExpertQA dataset.
  • Figure 4: Additional reward curves across different datasets and algorithms. For baseline models, we terminate the training process if an obvious convergence trend is observed.
  • Figure 5: Distribution of thinking-to-answer Coupling Ratio on ExpertQA. The red curve (Qwen3-1.7B) exhibits a sharp peak at $Ratio \approx 0.65$.