API-guided Dataset Synthesis to Finetune Large Code Models

TL;DR

DataScope presents a two-component framework for API-guided dataset synthesis to improve LCM fine-tuning: Dsel selects compact, high-API-coverage subsets from existing data, while Dgen generates domain-specific data from scratch using API combinations and SKDSL. The approach yields superior or comparable model performance with far fewer examples, reducing training cost and reliance on large raw datasets. Extensive experiments across general and domain-specific settings demonstrate substantial gains in code generation tasks, with analyses of internal model behavior, hyperparameters, and cost-effectiveness supporting the efficacy of API-guided synthesis. This work highlights the practical value of API abstractions for constructing high-quality SFT datasets and enables scalable, targeted fine-tuning of large code models.

Abstract

Large code models (LCMs), pre-trained on vast code corpora, have demonstrated remarkable performance across a wide array of code-related tasks. Supervised fine-tuning (SFT) plays a vital role in aligning these models with specific requirements and enhancing their performance in particular domains. However, synthesizing high-quality SFT datasets poses a significant challenge due to the uneven quality of datasets and the scarcity of domain-specific datasets. Inspired by APIs as high-level abstractions of code that encapsulate rich semantic information in a concise structure, we propose DataScope, an API-guided dataset synthesis framework designed to enhance the SFT process for LCMs in both general and domain-specific scenarios. DataScope comprises two main components: Dsel and Dgen. On one hand, Dsel employs API coverage as a core metric, enabling efficient dataset synthesis in general scenarios by selecting subsets of existing (uneven-quality) datasets with higher API coverage. On the other hand, Dgen recasts domain dataset synthesis as a process of using API-specified high-level functionality and deliberately-constituted code skeletons to synthesize concrete code. Extensive experiments demonstrate DataScope's effectiveness, with models fine-tuned on its synthesized datasets outperforming those tuned on unoptimized datasets five times larger. Furthermore, a series of analyses on model internals, relevant hyperparameters, and case studies provide additional evidence for the efficacy of our proposed methods. These findings underscore the significance of dataset quality in SFT and advance the field of LCMs by providing an efficient, cost-effective framework for constructing high-quality datasets. This contribution enhances performance across both general and domain-specific scenarios, paving the way for more powerful and tailored LCMs.

Figures (7)

• Figure 1: Overview of the proposed API-guided dataset synthesis framework, from its (a) targeted scenarios to (b) methodology, and further to (c) usage for enhancing LCM SFT in general and domain-specific scenarios.
• Figure 2: An illustrative example showing the crucial role of APIs in helping LCMs understand programs. For readability, we have shortened the example code and simplified the API calls. The blue box on the left shows the original highly abstract code using APIs, while the right one displays the code after we manually expanded the API calls. The tokens highlighted in red are the top 20 tokens with the highest average attention scores across all attention heads in the model's last layer.
• Figure 3: Pass@1 score of Dsel, Random, and CAR on the Humaneval benchmark using CodeLlama models of different sizes trained on subsets of varying sizes from the OSS and CODEE datasets in Bench.
• Figure 4: Overview of Dgen for SFT dataset generation and Bench for evaluation, with an example of key components in Dgen's instruction generation process.
• Figure 5: (a) Clustering behavior in the 7B model. (b) and (c) Comparison of the base 13B model and model fine-tuned with Dgen in distinguishing between Matplotlib and Pandas.