Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models

TL;DR

This work investigates how supervised fine-tuning (SFT) and reinforcement learning (RL) interact in training code-focused LLMs by constructing a controlled synthetic dataset from handcrafted atomic Python functions. A data-synthesis pipeline generates composite problems, enabling rigorous ablations of SFT and RL across carefully designed datasets, with a sandboxed unit-test feedback loop. Key findings show that both atomic and synthetic data are essential for SFT generalization, a small number of composites can suffice, and RL—especially when initialized from an SFT checkpoint—substantially enhances target-domain generalization, while training RL from scratch can mitigate overfitting. The insights point toward cost-effective data curation and clear directions for scaling to broader languages and benchmarks.

Abstract

Automatic code generation has been a longstanding research topic. With the advancement of general-purpose large language models (LLMs), the ability to code stands out as one important measure to the model's reasoning performance. Usually, a two-stage training paradigm is implemented to obtain a Code LLM, namely the pretraining and the fine-tuning. Within the fine-tuning, supervised fine-tuning (SFT), and reinforcement learning (RL) are often used to improve the model's zero-shot ability. A large number of work has been conducted to improve the model's performance on code-related benchmarks with either modifications to the algorithm or refinement of the dataset. However, we still lack a deep insight into the correlation between SFT and RL. For instance, what kind of dataset should be used to ensure generalization, or what if we abandon the SFT phase in fine-tuning. In this work, we make an attempt to understand the correlation between SFT and RL. To facilitate our research, we manually craft 100 basis python functions, called atomic functions, and then a synthesizing pipeline is deployed to create a large number of synthetic functions on top of the atomic ones. In this manner, we ensure that the train and test sets remain distinct, preventing data contamination. Through comprehensive ablation study, we find: (1) Both atomic and synthetic functions are indispensable for SFT's generalization, and only a handful of synthetic functions are adequate; (2) Through RL, the SFT's generalization to target domain can be greatly enhanced, even with the same training prompts; (3) Training RL from scratch can alleviate the over-fitting issue introduced in the SFT phase.

Figures (21)

• Figure 1: Workflow of the data synthesizing process. In step 1, we manually craft 100 atomic python functions covering a few programming topics, such as math and sorting algorithms. In the second step, we run the data synthesizing pipeline and obtain a large number of synthetic functions. Finally, all the synthetic functions will be checked via the code sandbox for validity.
• Figure 2: SFT with different mixtures of Composite_A and Atom_base, where we randomly down-sample Composite_A while up-sample Atom_base.
• Figure 3: SFT with a mixture of Composite_A and Atom_modify. We randomly down-sample Composite_A to 1% while up-sample Atom_modify to 600%.
• Figure 4: SFT with different mixtures of Composite_A and Atom_translate, where we randomly down-sample Composite_A while up-sample Atom_translate.
• Figure 5: SFT with translated prompts, where en2cn denotes <prompt_English, prompt_Chinese> pairs, cn2en means <prompt_Chinese, prompt_English> pairs.