Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs

Xiaoxuan Wang, Bo Liu, Song Jiang, Jingzhou Liu, Jingyuan Qi, Xia Chen, Baosheng He

TL;DR

The paper tackles the limitation that LLMs struggle to verify their own reasoning traces. It introduces GRPO-Verif, a unified objective that jointly optimizes solution generation and self-verification using an auxiliary term weighted by $\alpha$, built on GRPO with group-normalized advantages. Empirical results on four math benchmarks show that GRPO improves verification, and GRPO-Verif further boosts verification accuracy to about $37.1\%$ while maintaining solution performance near $38.5\%$, demonstrating that explicit self-verification enhances reliability without sacrificing reasoning ability. The work suggests a practical path toward more verifiable and robust reasoning in LLMs, while noting computational overhead from generating and training on verifications as a future efficiency challenge.

Abstract

The reasoning capabilities of large language models (LLMs) have been significantly improved through reinforcement learning (RL). Nevertheless, LLMs still struggle to consistently verify their own reasoning traces. This raises the research question of how to enhance the self-verification ability of LLMs and whether such an ability can further improve reasoning performance. In this work, we propose GRPO-Verif, an algorithm that jointly optimizes solution generation and self-verification within a unified loss function, with an adjustable hyperparameter controlling the weight of the verification signal. Experimental results demonstrate that our method enhances self-verification capability while maintaining comparable performance in reasoning.

From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs

TL;DR

The paper tackles the limitation that LLMs struggle to verify their own reasoning traces. It introduces GRPO-Verif, a unified objective that jointly optimizes solution generation and self-verification using an auxiliary term weighted by , built on GRPO with group-normalized advantages. Empirical results on four math benchmarks show that GRPO improves verification, and GRPO-Verif further boosts verification accuracy to about while maintaining solution performance near , demonstrating that explicit self-verification enhances reliability without sacrificing reasoning ability. The work suggests a practical path toward more verifiable and robust reasoning in LLMs, while noting computational overhead from generating and training on verifications as a future efficiency challenge.

Abstract

The reasoning capabilities of large language models (LLMs) have been significantly improved through reinforcement learning (RL). Nevertheless, LLMs still struggle to consistently verify their own reasoning traces. This raises the research question of how to enhance the self-verification ability of LLMs and whether such an ability can further improve reasoning performance. In this work, we propose GRPO-Verif, an algorithm that jointly optimizes solution generation and self-verification within a unified loss function, with an adjustable hyperparameter controlling the weight of the verification signal. Experimental results demonstrate that our method enhances self-verification capability while maintaining comparable performance in reasoning.

Paper Structure

This paper contains 14 sections, 5 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminary: GRPO
  4. Self-Verification GRPO: GRPO-Verif
  5. Experiment
  6. Experiment Setup
  7. Experiment Result
  8. Related Work
  9. Conclusion
  10. Implementation Detail
  11. Reproducibility
  12. Reproducibility
  13. Prompt
  14. Limitation

Figures (1)

  • Figure 1: GRPO-Verif Overview. For each question, the policy model generates candidate solutions and computes advantages from solution rewards. Verification responses are then derived from the paired question–solution inputs, with advantages from verification rewards. Both sets of advantages contribute separately to the optimization and are combined into a joint loss to update the policy.