Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Likelihood-Based Reward Designs for General LLM Reasoning

Ariel Kwiatkowski, Natasha Butt, Ismail Labiad, Julia Kempe, Yann Ollivier

TL;DR

This work tackles the challenge of reward design for RL-based CoT fine-tuning of LLMs by evaluating likelihood-based rewards derived from the probability or log-probability of reference answers, including $R(z,a)$ forms such as $R(z,a)=\log \pi_\theta(a^*|p,z)$. It systematically compares log-probability rewards against binary and other probability-based signals across verifiable (MATH, DeepScaleR) and non-verifiable (Alpaca, NuminaProof) tasks, using two model families ($\text{Llama-}3.2$, $\text{Qwen-}2.5$). The key finding is that the log-probability of the reference answer for CoT training remains robust across settings, improving perplexity and, in verifiable domains, achieving comparable or superior success rates relative to standard RL, while other probability-based rewards can fail in non-verifiable tasks due to vanishing probabilities. This provides a unified, verifier-free training signal that bridges short, verifiable and long, non-verifiable reasoning, offering a practical pathway for general-purpose CoT RL fine-tuning with wide applicability.

Abstract

Fine-tuning large language models (LLMs) on reasoning benchmarks via reinforcement learning requires a specific reward function, often binary, for each benchmark. This comes with two potential limitations: the need to design the reward, and the potentially sparse nature of binary rewards. Here, we systematically investigate rewards derived from the probability or log-probability of emitting the reference answer (or any other prompt continuation present in the data), which have the advantage of not relying on specific verifiers and being available at scale. Several recent works have advocated for the use of similar rewards (e.g., VeriFree, JEPO, RLPR, NOVER). We systematically compare variants of likelihood-based rewards with standard baselines, testing performance both on standard mathematical reasoning benchmarks, and on long-form answers where no external verifier is available. We find that using the log-probability of the reference answer as the reward for chain-of-thought (CoT) learning is the only option that performs well in all setups. This reward is also consistent with the next-token log-likelihood loss used during pretraining. In verifiable settings, log-probability rewards bring comparable or better success rates than reinforcing with standard binary rewards, and yield much better perplexity. In non-verifiable settings, they perform on par with SFT. On the other hand, methods based on probability, such as VeriFree, flatline on non-verifiable settings due to vanishing probabilities of getting the correct answer. Overall, this establishes log-probability rewards as a viable method for CoT fine-tuning, bridging the short, verifiable and long, non-verifiable answer settings.

Likelihood-Based Reward Designs for General LLM Reasoning

TL;DR

This work tackles the challenge of reward design for RL-based CoT fine-tuning of LLMs by evaluating likelihood-based rewards derived from the probability or log-probability of reference answers, including forms such as . It systematically compares log-probability rewards against binary and other probability-based signals across verifiable (MATH, DeepScaleR) and non-verifiable (Alpaca, NuminaProof) tasks, using two model families (, ). The key finding is that the log-probability of the reference answer for CoT training remains robust across settings, improving perplexity and, in verifiable domains, achieving comparable or superior success rates relative to standard RL, while other probability-based rewards can fail in non-verifiable tasks due to vanishing probabilities. This provides a unified, verifier-free training signal that bridges short, verifiable and long, non-verifiable reasoning, offering a practical pathway for general-purpose CoT RL fine-tuning with wide applicability.

Abstract

Fine-tuning large language models (LLMs) on reasoning benchmarks via reinforcement learning requires a specific reward function, often binary, for each benchmark. This comes with two potential limitations: the need to design the reward, and the potentially sparse nature of binary rewards. Here, we systematically investigate rewards derived from the probability or log-probability of emitting the reference answer (or any other prompt continuation present in the data), which have the advantage of not relying on specific verifiers and being available at scale. Several recent works have advocated for the use of similar rewards (e.g., VeriFree, JEPO, RLPR, NOVER). We systematically compare variants of likelihood-based rewards with standard baselines, testing performance both on standard mathematical reasoning benchmarks, and on long-form answers where no external verifier is available. We find that using the log-probability of the reference answer as the reward for chain-of-thought (CoT) learning is the only option that performs well in all setups. This reward is also consistent with the next-token log-likelihood loss used during pretraining. In verifiable settings, log-probability rewards bring comparable or better success rates than reinforcing with standard binary rewards, and yield much better perplexity. In non-verifiable settings, they perform on par with SFT. On the other hand, methods based on probability, such as VeriFree, flatline on non-verifiable settings due to vanishing probabilities of getting the correct answer. Overall, this establishes log-probability rewards as a viable method for CoT fine-tuning, bridging the short, verifiable and long, non-verifiable answer settings.
Paper Structure (26 sections, 1 theorem, 11 equations, 22 figures, 3 tables)

This paper contains 26 sections, 1 theorem, 11 equations, 22 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Our Approach and Contributions.
  3. Related Work.
  4. Method
  5. Context: Chain-of-thought fine-tuning via Reinforcement Learning.
  6. RL fine-tuning with probability-based rewards.
  7. Algorithms and rewards tested.
  8. Success metrics.
  9. Experimental Results
  10. Setup: Datasets, Models, and Protocol
  11. Results on Verifiable Domains
  12. Results on Non-verifiable Domains
  13. Length of the Chain-of-Though During Training
  14. Discussion: Why don't CoTs improve performance for non-verifiable domains?
  15. Conclusion
  16. ...and 11 more sections

Key Result

Lemma 1

Let $z$ be a chain-of-thought variable sampled from a model $\pi_\theta$ with parameters $\theta$, and let $R_\theta(z)$ be a reward function that depends on $z$ and also possibly on $\pi_\theta$ (for instance, $R_\theta(z)=\log \pi_\theta(a^\star|z)$ or $R_\theta(z)=\pi_\theta(a^\star|z)$). Then th has the same gradients (up to sign) as the loss function where ${}{^\mathrm{sg}}$ denotes a stop-g

Figures (22)

  • Figure 1: Verifiable. Llama 3.2 3B Instruct on MATH, G=32. Learning curves of our algorithms for various metrics. Dashed curves represent the RL baseline and (no-CoT) SFT; green shades for the logprob family of rewards (Logprob, Average logprob and JEPO) and blue for probability-based rewards Probability (VeriFree) and Average Probability (RLPR). Numerical values can be found in \ref{['tab:verifiable_g32']}.
  • Figure 2: Non-verifiable: Qwen 2.5 3B Instruct on NuminaProof. Learning curves of our algorithms for three metrics. Numerical values can be found in \ref{['tab:nonverifiable_base']}. Log-prob family models match the (per-answer) average log-prob and perplexity from SFT, while probability rewards fail to improve on these metrics due to the sparsity of the rewards. We observe a rapid "collapse" in CoT-length for the log-prob family.
  • Figure 3: Verifiable. Qwen 2.5 3B Instruct on MATH with a group size of 32.
  • Figure 4: Verifiable. Llama 3.2 3B Instruct on DeepScaleR with a group size of 32.
  • Figure 5: Verifiable. Qwen 2.5 3B Instruct on DeepScaleR with a group size of 32.
  • ...and 17 more figures

Theorems & Definitions (2)

  • Lemma 1
  • proof