On the Effectiveness of Large Language Models in Domain-Specific Code Generation
Xiaodong Gu, Meng Chen, Yalan Lin, Yuhan Hu, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, Juhong Wang
TL;DR
The paper addresses the challenge of domain-specific code generation by revealing that standard LLMs underperform when writing domain-tailored code due to limited library knowledge. It proposes DomCoder, a three-pronged approach combining an external API inquirer (kg-GPT), chain-of-thought prompting, and chain-of-thought fine-tuning to inject domain knowledge during generation. Through a domain-specific dataset spanning web and game development libraries and across three backbones (ChatGPT, PolyCoder, CodeLlama), it demonstrates substantial improvements in BLEU and CodeBLEU, with zero-shot and fine-tuning variants performing best in respective settings. An in-house Tencent study corroborates real-world gains, suggesting that integrating API knowledge into prompts and training can meaningfully enhance domain-specific coding. The work outlines future directions to broaden knowledge sources and pretraining strategies to further close the gap between domain-specific and open-domain code generation.
Abstract
Large language models (LLMs) such as ChatGPT have shown remarkable capabilities in code generation. Despite significant achievements, they rely on enormous training data to acquire a broad spectrum of open-domain knowledge. Besides, their evaluation revolves around open-domain benchmarks like HumanEval, which primarily consist of programming contests. Therefore, it is hard to fully characterize the intricacies and challenges associated with particular domains (e.g., web, game, and math). In this paper, we conduct an in-depth study of the LLMs in domain-specific code generation. Our results demonstrate that LLMs exhibit sub-optimal performance in generating domain-specific code, due to their limited proficiency in utilizing domain-specific libraries. We further observe that incorporating API knowledge as prompts can empower LLMs to generate more professional code. Based on these findings, we further investigate how to effectively incorporate API knowledge into the code generation process. We experiment with three strategies for incorporating domain knowledge, namely, external knowledge inquirer, chain-of-thought prompting, and chain-of-thought fine-tuning. We refer to these strategies as a new code generation approach called DomCoder. Experimental results show that all strategies of DomCoder lead to improvement in the effectiveness of domain-specific code generation under certain settings.