Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

TL;DR

The paper tackles the problem that evaluating LLM code generation on real-world tasks is confounded by training-data leakage and code familiarity. It introduces ObfusEval, a code-obfuscation based benchmark built from 1,354 functions across five real-world C projects, applying symbol, structure, and semantic obfuscation to descriptions, code, and dependencies. By testing four LLMs with compile and test pass rates (CPR and TPR), the study shows that obfuscation can dramatically reduce performance (up to 62.5%), revealing more accurate limits of current models and stabilizing model rankings under unseen conditions. The work provides a practical framework for assessing true code-generation capabilities and highlights prevalent syntax and non-functional issues that persist even when generated code passes tests, informing future directions for evaluation and model development.

Abstract

Recently, large language models (LLMs) have shown strong potential in code generation tasks. However, there are still gaps before they can be fully applied in actual software development processes. Accurately assessing the code generation capabilities of large language models has become an important basis for evaluating and improving the models. Some existing works have constructed datasets to evaluate the capabilities of these models. However, the current evaluation process may encounter the illusion of "Specialist in Familiarity", primarily due to three gaps: the exposure of target code, case timeliness, and dependency availability. The fundamental reason for these gaps is that the code in current datasets may have been extensively exposed and exercised during the training phase, and due to the continuous training and development of LLM, their timeliness has been severely compromised. The key to solve the problem is to, as much as possible, evaluate the LLMs using code that they have not encountered before. Thus, the fundamental idea in this paper is to draw on the concept of code obfuscation, changing code at different levels while ensuring the functionality and output. To this end, we build a code-obfuscation based benchmark OBFUSEVAL. We first collect 1,354 raw cases from five real-world projects, including function description and code. Then we use three-level strategy (symbol, structure and semantic) to obfuscate descriptions, code and context dependencies. We evaluate four LLMs on OBFU- SEVAL and compared the effectiveness of different obfuscation strategy. We use official test suites of these projects to evaluate the generated code. The results show that after obfuscation, the average decrease ratio of test pass rate can up to 62.5%.

Figures (6)

• Figure 1: Examples of code obfuscation.
• Figure 2: Workflow of dataset construction and testing
• Figure 3: An example of code obfuscation process
• Figure 4: Example of prompt for code generation scenarios
• Figure 5: TPR under different code obfuscation strategies. After introducing various degrees of obfuscation, the rankings of four LLMS tend to be stabilize and consistent.