ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?

TL;DR

ECCO presents a reproducible benchmark to study code efficiency without sacrificing functional correctness, addressing instability in hardware-based performance measurements. It evaluates two generation/editing paradigms (history-based editing and NL-instructed generation) under three optimization strategies (in-context learning, iterative refinement, and fine-tuning) using a cloud Judge0 platform and a large CodeNet-derived Python problem set. Key findings show that execution feedback and targeted fine-tuning help maintain correctness, while NL feedback often improves efficiency, yet no method consistently achieves faster runtime or smaller memory usage without harming correctness. The ECCO framework and results provide a solid foundation for future research on correctness-preserving program optimizations and benchmarking across languages.

Abstract

Although large language models (LLMs) have been largely successful in generating functionally correct programs, conditioning models to produce efficient solutions while ensuring correctness remains a challenge. Further, unreliability in benchmarking code efficiency is a hurdle across varying hardware specifications for popular interpreted languages such as Python. In this paper, we present ECCO, a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. On ECCO, we adapt and thoroughly investigate the three most promising existing LLM-based approaches: in-context learning, iterative refinement with execution or NL feedback, and fine-tuning conditioned on execution and editing history. While most methods degrade functional correctness and moderately increase program efficiency, we find that adding execution information often helps maintain functional correctness, and NL feedback enhances more on efficiency. We release our benchmark to support future work on LLM-based generation of efficient code.

Figures (14)

• Figure 1: Correctness-preserving versus spurious optimization when optimizing a linear search algorithm with binary search on a sorted list. Spurious optimization can reduce runtime, but add errors that cause the program to be incorrect. In contrast, a true optimization reduces runtime and remains correct, as we emphasized in ECCO.
• Figure 2: Evaluation platform using Judge0.
• Figure 3: Illustration of history-based editing.
• Figure 4: Illustration of NL-instructed generation.
• Figure 5: Iterative refinement methods utilizing different forms of feedback. Self-Refine uses Natural Language feedback, Exec-Refine uses raw execution results on $T_{public}$ and NL+Exec-Refine uses NL reflection of execution.