From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones

TL;DR

This paper introduces a controlled synthetic framework to study whether reinforcement learning (RL) can induce genuinely new compositional skills in large language models, beyond reweighting existing reasoning. A two-stage training protocol separates atomic skill acquisition from compositional skill learning, and rigorous held-out evaluation tests generalization to deeper compositions and cross-task transfer. The authors show that RL, when incentivized to compose atomic skills, yields new compositional capabilities that transfer to different domains and reveal distinct failure modalities, unlike rejection-finetuning alone. These findings suggest that RL can meaningfully expand LLM capabilities, particularly for hard, compositional reasoning tasks, and highlight the importance of task design and incentivization in post-training approaches.

Abstract

Does RL teach LLMs genuinely new skills, or does it merely activate existing ones? This question lies at the core of ongoing debates about the role of RL in LLM post-training. On one side, strong empirical results can be achieved with RL even without preceding supervised finetuning; on the other, critics argue that RL contributes little beyond reweighting existing reasoning strategies. This work provides concrete evidence that LLMs can acquire genuinely new skills during RL by composing existing ones, mirroring one of the central mechanisms by which humans acquire new cognitive skills. To mitigate data contamination and other confounding factors, and to allow precise control over task complexity, we develop a synthetic framework for our investigation. Specifically, we define a skill as the ability to infer the output of a string transformation function f(x) given x. When an LLM has already learned f and g prior to RL, our experiments reveal that RL enables it to learn unseen compositions of them h(x)=g(f(x)). Further, this compositional ability generalizes to more difficult problems such as compositions of >2 functions unseen during RL training. Surprisingly, our experiments show that compositional skill acquired on a source task transfers to a different target task. This transfer happens even without compositional training on the target, requiring only prior knowledge of the target's atomic skills. Our qualitative analysis shows that RL fundamentally changes the reasoning behaviors of the models. In contrast, next-token training with the same data yields none of these findings. Our systematic experiments provide fresh insights into LLM learning, suggesting the value of first building base models with basic skills, then using RL to incentivize advanced, generalizable skills for complex problems.

Figures (9)

• Figure 1: An overview of our research framework and key findings. (Top) We introduce a clean string transformation testbed to scientifically analyze RL's capabilities. (Bottom-Left) Our central RL Compositionality Hypothesis posits that training on simple composites with RL unlocks generalizable compositional skills. (Bottom-Right) Our experiments validate this, showing that: (1) compositional data combined with RL is the key ingredient for learning this new skill; (2) the learned skill transfers across domains; and (3) RL significantly improves difficult problems where the base model fails, while only reranking on problems it solves well.
• Figure 2: Test Accuracy on held-out tasks vs. RL training steps, each related to one held-out task difficulty level. The dark blue curve indicates that training on atomic skills alone (RL Level 1) yields nearly no compositional ability on held-out functions. In contrast, including Level 2 data in RL unlocks strong generalization to more complex problems (Levels 3-6).
• Figure 3: RL vs. RFT on Compositional Tasks. RL (red dashed line) achieves substantially higher accuracy across all levels, while iterative RFT fails to learn a generalizable skill.
• Figure 4: Avg@32 Accuracy on the Countdown Task. Atomic skills are a prerequisite for task transfer, and that compositional RL (Multi-Base + RL L1+2) on the unrelated string task offers a significant performance improvement on Countdown. Note that none of the models are trained with RL on Countdown.
• Figure 5: Pass@$k$ performance across varying difficulty levels. On easy problems (Levels 1-2), the performance gap shrinks with more samples, consistent with the reranking narrative. On hard problems (Levels 3-8), the gap widens substantially, suggesting new skill acquisition.