Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency

TL;DR

Code-Optimise tackles the dual objective of code correctness and runtime efficiency by generating self-supervised preference data and training with Direct Preference Optimisation. It builds a lightweight, on-policy pipeline (Sampling, Annotation, Optimisation) and uses Dynamic Solution Selection to reduce overfitting without needing proprietary data. Across MBPP and HumanEval, DPO improves $pass@k$, reduces runtime by up to 6% (in-domain MBPP) and 3% (out-of-domain), and shortens generated code by up to 48% (MBPP) and 23% (HumanEval). This work demonstrates a data-efficient route to multi-objective code generation and highlights the importance of training signal design and solution selection for robust, efficient code synthesis.

Abstract

Code Language Models have been trained to generate accurate solutions, typically with no regard for runtime. On the other hand, previous works that explored execution optimisation have observed corresponding drops in functional correctness. To that end, we introduce Code-Optimise, a framework that incorporates both correctness (passed, failed) and runtime (quick, slow) as learning signals via self-generated preference data. Our framework is both lightweight and robust as it dynamically selects solutions to reduce overfitting while avoiding a reliance on larger models for learning signals. Code-Optimise achieves significant improvements in pass@k while decreasing the competitive baseline runtimes by an additional 6% for in-domain data and up to 3% for out-of-domain data. As a by-product, the average length of the generated solutions is reduced by up to 48% on MBPP and 23% on HumanEval, resulting in faster and cheaper inference. The generated data and codebase is open-sourced at https://github.com/huawei-noah/HEBO/tree/Code_Optimise.

Figures (12)

• Figure 1: Overview of Code-Optimise. (1) Diverse solutions are sampled per problem. (2) A code interpreter annotates the solutions by functional correctness and runtime. (3) CLM is optimised using SFT or DPO.
• Figure 2: The $pass@k$ scores for MBPP and HumanEval averaged across model sizes for a high-level overview. Models optimised via DPO consistently show higher functional correctness compared to Base and SFT for all $k$.
• Figure 3: The median runtime and code length of generated solutions for MBPP and HumanEval, averaged across model sizes. Values shown are the percentage changes relative to Base, i.e. >0 is slower or longer than Base, <0 is faster or shorter. The best DPO models achieve a reduced runtime compared to SFT models as well as the very competitive Base models. A significant reduction in code length (10% - 20%) is observed across both datasets.
• Figure 4: The $pass@1$, $pass@10$ and $pass@100$ scores for MBPP and HumanEval as the number of parameters increases. A significant improvement over competitive Base and SFT models can be observed for DPO configs.
• Figure 5: The runtimes for MBPP and HumanEval as model size increases. Values shown are the percentage changes relative to Base, i.e. >0 means slower than Base, <0 means faster. On average, DPO models show a greater runtime reduction on in-domain rather than out-of-domain data. SFT models exhibit inconsistent scaling patterns.