CRPE: Expanding The Reasoning Capability of Large Language Model for Code Generation

TL;DR

CRPE tackles the challenge of enabling LLMs to perform deeper code reasoning by proposing a three-stage data synthesis and training framework. It leverages a multi-agent Code-COT Maker to produce high-quality reasoning data and employs a tree-search based self-improvement with Step-DPO to refine reasoning paths. The approach yields superior results on LiveCodeBench with 7B and 32B variants surpassing several baselines, and competitive performance on HumanEval and MBPP after refinement. The work contributes an open-source data pipeline for code problems and a reasoning-data-centric finetuning paradigm that can enhance practical code generation tasks.

Abstract

We introduce CRPE (Code Reasoning Process Enhancer), an innovative three-stage framework for data synthesis and model training that advances the development of sophisticated code reasoning capabilities in large language models (LLMs). Building upon existing system-1 models, CRPE addresses the fundamental challenge of enhancing LLMs' analytical and logical processing in code generation tasks. Our framework presents a methodologically rigorous yet implementable approach to cultivating advanced code reasoning abilities in language models. Through the implementation of CRPE, we successfully develop an enhanced COT-Coder that demonstrates marked improvements in code generation tasks. Evaluation results on LiveCodeBench (20240701-20240901) demonstrate that our COT-Coder-7B-StepDPO, derived from Qwen2.5-Coder-7B-Base, with a pass@1 accuracy of 21.88, exceeds all models with similar or even larger sizes. Furthermore, our COT-Coder-32B-StepDPO, based on Qwen2.5-Coder-32B-Base, exhibits superior performance with a pass@1 accuracy of 35.08, outperforming GPT4O on the benchmark. Overall, CRPE represents a comprehensive, open-source method that encompasses the complete pipeline from instruction data acquisition through expert code reasoning data synthesis, culminating in an autonomous reasoning enhancement mechanism.

Figures (12)

• Figure 1: The overview of CRPE, which comprises three steps: (i) synthesizing large-scale code problems and (ii) synthesizing high-quality code reasoning datas for supervised finetuning and (iii) prompting code reasoning model derived from the second step to sample reasoning paths and achieve self-improve.
• Figure 2: Code Problem Pipeline.
• Figure 3: The Thinking Agent Illustration.
• Figure 4: The Reflection Agent Illustration.
• Figure 5: The Execution Agent Illustration.