Knowledge-Aware Code Generation with Large Language Models

TL;DR

The paper tackles the challenge of large language models solving novel programming problems while avoiding training-data leakage by introducing the CodeF dataset and a knowledge-aware code generation framework called KareCoder. KareCoder combines a Knowledge Library with a two-stage process (Prompt Engineering and Coding) to generate knowledge-aware prompts that guide code production, achieving a notable improvement in Pass@1 on CodeF post-2021-09 and competitive results on APPS. The work also analyzes different knowledge representations, shot settings, and dataset effects, finding that the Knowledge Description format most effectively enhances generalization to unseen problems. Overall, the approach demonstrates the practical value of external structured programming knowledge to enhance LLM-based code generation, offering a path toward more robust, knowledge-guided problem solving in software tasks.

Abstract

Large Language Models (LLMs) perform well on basic programming problems. However, they encounter challenges when dealing with complex tasks involving the use of diverse algorithmic and data structure skills, particularly programming competition-level problems. Notably, ChatGPT exhibits proficient performance on problems it has encountered during its pre-training phase, but this performance deteriorates when faced with novel problems. Consequently, enhancing the ability of LLMs to address unfamiliar problems has emerged as a pivotal research focus. The problem-solving process of LLMs mirrors human programmers' approach to a certain extent. When confronted with new programming tasks, human programmers engage in task planning and code writing with the previously acquired knowledge about algorithms and data structures. Despite having learned such knowledge, LLMs struggle to effectively apply it when faced with specific new problems. To address this issue, we constructed a novel dataset, CodeF, which contains a portion of programming problems that ChatGPT has not previously encountered. Furthermore, we developed a Knowledge Library tailored for Python programming contest problems and introduced the concept of Knowledge-Aware Code Generation (KareCoder). KareCoder bolsters the models' understanding and problem-solving capabilities by integrating prompt and knowledge from the library into the LLMs' code generation reasoning process, especially on Pass@1 metrics. Upon testing on the CodeF and APPS datasets, KareCoder demonstrated outstanding performance in handling novel problems previously unencountered by LLMs. In contrast with the code directly generated by ChatGPT, KareCoder achieved a relative improvement of 23.3% on the Pass@1 metric on the CodeF post2021-9 dataset. Additionally, it performs well compared to other methods when dealing with problems that LLMs have previously encountered.

Figures (7)

• Figure 1: Experimental validation of split-difficulties on CodeF post2021-9 and CodeF pre2021-9 datasets.
• Figure 2: Schematic diagram of CodeF acquisition and processing. The processing part includes data cleaning and splitting.
• Figure 3: Distribution of algorithms and data structure tags.
• Figure 4: Examples of different Knowledge Libraries. The corresponding Greedy algorithm is shown here.
• Figure 5: An overview of KareCoder. Given a programming problem, KareCoder matches the Knowledge Library appropriate to the problem and then generates a prompt for solving the problem. Finally, the prompt is used to guide the generation of code.