Measuring Code Efficiency Optimization Capabilities with ACEOB

TL;DR

This work addresses the challenge of measuring and achieving code efficiency optimization in AI-assisted programming by introducing the IC2EC task and the ACEOB benchmark, specifically targeting Python. It defines two metrics, IOCCB and NPI, designed to evaluate efficiency independently of full functional correctness and to quantify optimization potential within problem-specific contexts. ACEOB also provides Python cost models (ACEOB-NPI) and includes three datasets (ACEOB-Ori, ACEOB-NPI, ACEOB) to support ground-truth pairing, time evaluation, and training of cost models. Experimental results show that applying an NPI filter improves model performance across encoder–decoder and decoder-only architectures, though state-of-the-art models like ChatGPT still struggle with practical code efficiency optimization, highlighting the need for specialized efficiency-focused data and tools. Overall, ACEOB offers a rigorous, scalable framework for advancing automatic generation of high-performance code and reducing resource consumption in AI-assisted programming.

Abstract

As Moore's Law gains diminish, software performance and efficiency become increasingly vital. Optimizing code efficiency is challenging, even for professional programmers. However, related research remains relatively scarce, and rigorously assessing models' abilities to optimize code efficiency is fraught with difficulties. In response to this challenge, we first conduct an in-depth analysis of "code patterns" in the model training dataset, meticulously exploring human-written code. Secondly, we define a task for optimizing code efficiency and introduce the Automatic Code Efficiency Optimization Benchmark (ACEOB), which consists of 95,359 pairs of efficient-inefficient code aimed at assessing code efficiency optimization capabilities. To our knowledge, ACEOB is the first dataset specifically targeting Python code efficiency optimization. To evaluate models' ability in optimizing code efficiency, we propose two new metrics: the Isomorphic Optimal Comparison CodeBLEU (IOCCB) metric and the Normalized Performance Index (NPI) metric, to assess the efficiency of model-generated code. We also evaluate several advanced code models, such as PolyCoder and CodeT5, after fine-tuning them on ACEOB and demonstrate that the efficiency of each model improves after introducing the NPI filter. However, it was observed that even ChatGPT does not perform optimally in code efficiency optimization tasks.

Figures (16)

• Figure 1: The problem difficulty and the weighting of each code efficiency.
• Figure 2: The problem difficulty and the weighting of each code efficiency.
• Figure 3: Distribution proportions of the execution time of code (top left), token length (top right), problem-solving difficulty (bottom left), and algorithm tags (bottom right).
• Figure 4: Examples of the IC2EC task. Each line corresponds to an efficient-inefficient code pair, containing inefficient and efficient code. The three examples respectively emphasize the speed advantage of using built-in libraries (top), reducing the number of API calls (middle), and adopting more efficient data structures and algorithms (bottom).
• Figure 5: The distribution of algorithm tags is presented within the 9,415 samples of the test set derived from the ACEOB dataset.