CoCoST: Automatic Complex Code Generation with Online Searching and Correctness Testing

TL;DR

The CoCoST framework is introduced, which enhances complex code generation by online searching for more information with planned queries and correctness testing for code refinement and serializes the complex inputs and outputs to improve comprehension.

Abstract

Large Language Models have revolutionized code generation ability by converting natural language descriptions into executable code. However, generating complex code within real-world scenarios remains challenging due to intricate structures, subtle bugs, understanding of advanced data types, and lack of supplementary contents. To address these challenges, we introduce the CoCoST framework, which enhances complex code generation by online searching for more information with planned queries and correctness testing for code refinement. Moreover, CoCoST serializes the complex inputs and outputs to improve comprehension and generates test cases to ensure the adaptability for real-world applications. CoCoST is validated through rigorous experiments on the DS-1000 and ClassEval datasets. Experimental results show that CoCoST substantially improves the quality of complex code generation, highlighting its potential to enhance the practicality of LLMs in generating complex code.

Figures (10)

• Figure 1: An Example of the Human Developer Code-writing Process Imitated by the CoCoST. After the problem is received, an online search is performed to simulate search results and create an initial version of the code. Test cases are then generated, and the code is executed to produce output results. The code is refined based on the correctness of these results.
• Figure 2: The Pipline of CoCoST. Step 1: LLM is employed to strategize the problem and formulate queries based on the outlined steps. These queries enable the retrieval of various information from the internet. A high-quality initial code can be obtained through effective planning and leveraging internet information. Step 2: LLM generates test cases to test the correctness of the initial code. The serialization of test results serves as crucial input for the subsequent cycle of code refinement. Through iterative refinement processes, the quality of the initial code can be significantly improved.
• Figure 3: Case Study for Correctness Testing.
• Figure 4: Case Study for Online Retrieval.
• Figure 5: Plan and Queries Generation Prompt on DS-1000.