DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation

TL;DR

This work tackles the problem of unreliable code-generation evaluation due to static benchmarks and data contamination by introducing DynaCode, a dynamic, complexity-aware benchmark that jointly considers code complexity via cyclomatic metrics and call-graph complexity through nested, executable structures. By generating up to 189 million unique tasks across 4 code-complexity units and 16 call-graph patterns, DynaCode reveals substantial performance drops for contemporary LLMs as task complexity increases, underscoring the framework's ability to differentiate model capabilities. The authors demonstrate that including call-graph structures yields deeper insights into LLM behavior, particularly subfunction interactions in nested code, and show that DynaCode mitigates memorization effects compared to static benchmarks. The benchmark, along with evaluation code, is publicly available, offering a scalable and reliable platform for rigorous code-generation assessment and future expansion to richer graph structures and languages.

Abstract

The rapid advancement of large language models (LLMs) has significantly improved their performance in code generation tasks. However, existing code benchmarks remain static, consisting of fixed datasets with predefined problems. This makes them vulnerable to memorization during training, where LLMs recall specific test cases instead of generalizing to new problems, leading to data contamination and unreliable evaluation results. To address these issues, we introduce DynaCode, a dynamic, complexity-aware benchmark that overcomes the limitations of static datasets. DynaCode evaluates LLMs systematically using a complexity-aware metric, incorporating both code complexity and call-graph structures. DynaCode achieves large-scale diversity, generating up to 189 million unique nested code problems across four distinct levels of code complexity, referred to as units, and 16 types of call graphs. Results on 12 latest LLMs show an average performance drop of 16.8% to 45.7% compared to MBPP+, a static code generation benchmark, with performance progressively decreasing as complexity increases. This demonstrates DynaCode's ability to effectively differentiate LLMs. Additionally, by leveraging call graphs, we gain insights into LLM behavior, particularly their preference for handling subfunction interactions within nested code. Our benchmark and evaluation code are available at https://github.com/HWH-2000/DynaCode.

Figures (13)

• Figure 1: Data contamination on the popular benchmarks MBPP and MBPP+. Meta-Llama-3-8B-Instruct exhibits a significant performance drop from MBPP and MBPP+ to DynaCode.
• Figure 2: Overview of our proposed DynaCode. (a) Classification of code complexity, resulting in code problems with varying levels of complexity. (b) Construction of function call graphs, categorized based on graph features. (c) Integration of code complexity and graph complexity to form two-dimensional complexity-aware metrics. (d) Benchmark generation process.
• Figure 3: Comparison of Average Pass@1 scores across 4 LLMs (GPT-4o, GPT-3.5-Turbo, WizardLM-2-8x22B, Meta-Llama-3.1-405B-Instruct) at different complexity levels.
• Figure 4: Experiment on the number of problems. For each graph in every unit, a corresponding number of problems is generated.
• Figure 5: Pass@1 scores of GPT-3.5-Turbo and Meta-Llama-3.1-8B-Instruct before and after fine-tuning on MBPP+ and DynaCode, and DynaCode unit functions.