Generalization of RLVR Using Causal Reasoning as a Testbed

TL;DR

The paper investigates how reinforcement learning with verifiable rewards (RLVR) generalizes for causal reasoning tasks by constructing a controlled causal-inference benchmark (RLCausal) with fully specified SCMs and three query levels.It compares RLVR to supervised fine-tuning (SFT) across model scales (3B, 7B, 32B) and training levels, showing RLVR provides stronger within- and across-level generalization only when the base model has sufficient initial reasoning competence, and when trained on higher-level or more complex queries.The analysis reveals RLVR improves marginalization strategy and reduces probability-derivation errors, with benefits scaling with model size and reasoning prior; counterfactual reasoning remains the hardest area and is not reliably solved in the tested settings.The work highlights the importance of initial reasoning ability and on-policy data collection for RLVR effectiveness, and proposes future work on execution vs strategy quality and broader causal-reasoning tasks.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for post-training large language models (LLMs) on complex reasoning tasks. Yet, the conditions under which RLVR yields robust generalization remain poorly understood. This paper provides an empirical study of RLVR generalization in the setting of probabilistic inference over causal graphical models. This setting offers two natural axes along which to examine generalization: (i) the level of the probabilistic query -- associational, interventional, or counterfactual -- and (ii) the structural complexity of the query, measured by the size of its relevant subgraph. We construct datasets of causal graphs and queries spanning these difficulty axes and fine-tune Qwen-2.5-Instruct models using RLVR or supervised fine-tuning (SFT). We vary both the model scale (3B-32B) and the query level included in training. We find that RLVR yields stronger within-level and across-level generalization than SFT, but only for specific combinations of model size and training query level. Further analysis shows that RLVR's effectiveness depends on the model's initial reasoning competence. With sufficient initial competence, RLVR improves an LLM's marginalization strategy and reduces errors in intermediate probability calculations, producing substantial accuracy gains, particularly on more complex queries. These findings show that RLVR can improve specific causal reasoning subskills, with its benefits emerging only when the model has sufficient initial competence.

Figures (29)

• Figure 1: Top: Our causal inference task for investigating generalization of RLVR (see \ref{['sec:setup']}), system prompt (\ref{['fig:system_prompt']}) omitted for space. Bottom Left: Generative process for sampling task instances, and solver for computing the reference (see \ref{['sec:data']}). Bottom Right: We generate association, intervention, and counterfactual queries to study RLVR's within-/across-level generalization.
• Figure 2: Illustration of graph modifications corresponding to each query level and its relevant (solid) and irrelevant subgraph (dashed). $\times$ denotes dependencies removed due to an intervention. Relevant nodes are defined as ancestors of either the observation or the query variable, after graph modifications are performed to account for any interventions. Left: original graph. Mid-left: 3 relevant nodes for association query $p({\textnormal{v}}_2\mid {\textnormal{v}}_5= v_5)$. Mid-right: 2 relevant nodes for intervention query $p({{\textnormal{v}}_5}_{({\textnormal{v}}_2=c)})$. Right: 11 relevant nodes for counterfactual query $p({{\textnormal{v}}_3}_{({\textnormal{v}}_2=c)}\mid {\textnormal{v}}_2=v_2)$.
• Figure 3: The algorithm with higher within-/across-level accuracy for different sizes and query types. Significant cells (paired-perm test at $p < 0.05$) bolded and colored.
• Figure 4: Top: Accuracy (y-axis) vs. LLM size (x-axis) when evaluated on intervention (left), association (middle), and counterfactual (right) queries. Red curves correspond to RLVR, blue curves correspond to SFT. Solid (-) curves are LLMs fine-tuned on the same level as evaluation, dashed (--) curves are trained on a different level from evaluation. Bottom: Reasoning (RLVR) vs non-reasoning (SFT) strategies, before and after fine-tuning. As scale increases, both reasoning and non-reasoning prior improve, though the reasoning prior benefits more from scaling.
• Figure 5: LLM judge (o4-mini) analysis of the marginalization strategy (top) and the existence of derivation errors (bottom) before and after RLVR. Derivation errors and marginalization strategies are annotated on (the same) 80 samples per level. Judge prompts (including category definitions) are included in \ref{['fig:llm_judge_prompt']}. Example traces of marginalization strategy are included in \ref{['fig:incremental', 'fig:brute', 'fig:immediate', 'fig:no_marg']}.