Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Is Compositional Reasoning Learnable from Verifiable Rewards?

Daniel Barzilai, Yotam Wolf, Ronen Basri

TL;DR

This work addresses whether outcome-level feedback can reliably induce correct intermediate reasoning in autoregressive models by introducing the task-advantage ratio, which quantifies how choosing a given intermediate task affects the likelihood of final verification success. It derives an exact expression for the expected per-step update under RLVR and shows that a uniform positive task-advantage guarantees recovery of the correct chain-of-thought with iteration complexity scaling as $O(S^2)$; without such structure, RLVR may converge to suboptimal compositions. The analysis also reveals that the base model quality, via the initial probability of selecting the correct task $p_0$, governs whether RLVR converges to optimal behavior, with a threshold at $p_0=1/3$. Together, these results provide a principled understanding of when RLVR succeeds or fails, highlighting the importance of problem structure and base-model strength for verifier-based reinforcement signals in reasoning tasks.

Abstract

The emergence of compositional reasoning in large language models through reinforcement learning with verifiable rewards (RLVR) has been a key driver of recent empirical successes. Despite this progress, it remains unclear which compositional problems are learnable in this setting using outcome-level feedback alone. In this work, we theoretically study the learnability of compositional problems in autoregressive models under RLVR training. We identify a quantity that we call the task-advantage ratio, a joint property of the compositional problem and the base model, that characterizes which tasks and compositions are learnable from outcome-level feedback. On the positive side, using this characterization, we show that compositional problems where correct intermediate steps provide a clear advantage are efficiently learnable with RLVR. We also analyze how such an advantage naturally arises in different problems. On the negative side, when the structural advantage is not present, RLVR may converge to suboptimal compositions. We prove that, in some cases, the quality of the base model determines if such an advantage exists and whether RLVR will converge to a suboptimal solution. We hope our analysis can provide a principled theoretical understanding of when and why RLVR succeeds and when it does not.

When Is Compositional Reasoning Learnable from Verifiable Rewards?

TL;DR

This work addresses whether outcome-level feedback can reliably induce correct intermediate reasoning in autoregressive models by introducing the task-advantage ratio, which quantifies how choosing a given intermediate task affects the likelihood of final verification success. It derives an exact expression for the expected per-step update under RLVR and shows that a uniform positive task-advantage guarantees recovery of the correct chain-of-thought with iteration complexity scaling as ; without such structure, RLVR may converge to suboptimal compositions. The analysis also reveals that the base model quality, via the initial probability of selecting the correct task , governs whether RLVR converges to optimal behavior, with a threshold at . Together, these results provide a principled understanding of when RLVR succeeds or fails, highlighting the importance of problem structure and base-model strength for verifier-based reinforcement signals in reasoning tasks.

Abstract

The emergence of compositional reasoning in large language models through reinforcement learning with verifiable rewards (RLVR) has been a key driver of recent empirical successes. Despite this progress, it remains unclear which compositional problems are learnable in this setting using outcome-level feedback alone. In this work, we theoretically study the learnability of compositional problems in autoregressive models under RLVR training. We identify a quantity that we call the task-advantage ratio, a joint property of the compositional problem and the base model, that characterizes which tasks and compositions are learnable from outcome-level feedback. On the positive side, using this characterization, we show that compositional problems where correct intermediate steps provide a clear advantage are efficiently learnable with RLVR. We also analyze how such an advantage naturally arises in different problems. On the negative side, when the structural advantage is not present, RLVR may converge to suboptimal compositions. We prove that, in some cases, the quality of the base model determines if such an advantage exists and whether RLVR will converge to a suboptimal solution. We hope our analysis can provide a principled theoretical understanding of when and why RLVR succeeds and when it does not.
Paper Structure (31 sections, 18 theorems, 102 equations, 4 figures)

This paper contains 31 sections, 18 theorems, 102 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. RLVR and LLM reasoning.
  5. Learnability of autoregressive compositional problems.
  6. Other benefits of Chain of Thought.
  7. Learning Compositions Without Supervision
  8. Setting and Preliminaries
  9. Autoregressive generation.
  10. Model structure.
  11. Compositional target and learning objective.
  12. Unsupervised post-training.
  13. Assumptions.
  14. Main Results: Learning Compositions With RLVR
  15. Successfully Learning Compositions
  16. ...and 16 more sections

Key Result

Theorem 5.2

Under assumptions ass: value-ass: incoherence, for any CoT step $s\in[S]$, time $t \in \mathbb{N} \cup \{0\}$, task $j\in[J]$ and new sample $\mathbf{x}_{s-1}$ that contains a prompt plus $s-1$ additional CoT steps, then if $\rho_{s, j}^{(t)} > 0$, the expectation of the logit updates (with respect Otherwise, if $\rho_{s, j}^{(t)} = 0$, then the expected value is equal $- \eta \gamma^2\mathbb{P}_

Figures (4)

  • Figure 1: Long addition as an autoregressive compositional problem. LLMs decompose the computation by using their chain of thought as a scratchpad.
  • Figure 2: RLVR as sequential task selection. An autoregressive model chooses among tasks at each step, each deterministically producing the next token. Only the final output is verified, and intermediate decisions receive no direct supervision. Dark arrows indicate the correct task composition.
  • Figure 3: Iteration Complexity needed to reach an accuracy of $0.9$ as a function of the CoT length (log-log scale). Problems with an inductive structure have a constant task advantage, and the iteration complexity scales like $S^2$. Without such a structure (e.g., solving parities), the advantage may decrease exponentially in $S$, and the iteration complexity becomes exponentially large.
  • Figure 4: How the base model can impact RLVR: We consider a base model whose "quality" is determined by $p_0$, the probability of picking the correct tasks at initialization. Within the setting defined in Sec. \ref{['sec: neg']}, the model converges to a suboptimal solution whenever $p_0<\frac{1}{3}$ (with $\Pr_t(V=1) \to 1/2$) but to the optimal solution whenever $p_0 > 1/3$. This is aligned with the task-advantage ratio $\rho_{s,1}$ which we show is $>1$ when $p_0>1/3$ and $<1$ when $p_0<1/3$. At each step, samples are drawn i.i.d. with a batch size of $256$, and the accuracy is computed with respect to the last batch.

Theorems & Definitions (34)

  • Definition 3.1: Autoregressive Compositions
  • Definition 5.1: task-advantage ratio
  • Theorem 5.2
  • Theorem 5.4
  • Proposition 6.1
  • Proposition 6.2
  • Proposition 6.3
  • Lemma 1.1
  • proof
  • Lemma 1.2
  • ...and 24 more