An Experimental Study of Real-Life LLM-Proposed Performance Improvements

TL;DR

This study assesses whether modern LLMs can generate patches that meaningfully improve performance in real-world Java software. It introduces PerfOpt, a dataset of 65 performance-focused tasks from four OSS projects, and an automated pipeline that uses two leading LLMs with four prompting strategies to produce JSON patches, validated via JMH benchmarks. Across extensive benchmarking, LLM patches often improve performance over the baseline, but human-written developer patches consistently outperform LLM-generated ones, highlighting the value of deep performance reasoning and context. The work positions LLMs as useful performance co-pilots that can brainstorm or propose alternative optimizations, provided they are integrated with rigorous repair, validation, and benchmarking workflows, and it points to future directions in dataset expansion, multi-agent optimization, and IDE/CI integration.

Abstract

Large Language Models (LLMs) can generate code, but can they generate fast code? In this paper, we study this question using a dataset of 65 real-world tasks mined from open-source Java programs. We specifically select tasks where developers achieved significant speedups, and employ an automated pipeline to generate patches for these issues using two leading LLMs under four prompt variations. By rigorously benchmarking the results against the baseline and human-authored solutions, we demonstrate that LLM-generated code indeed improves performance over the baseline in most cases. However, patches proposed by human developers outperform LLM fixes by a statistically significant margin, indicating that LLMs often fall short of finding truly optimal solutions. We further find that LLM solutions are semantically identical or similar to the developer optimization idea in approximately two-thirds of cases, whereas they propose a more original idea in the remaining one-third. However, these original ideas only occasionally yield substantial performance gains.

Figures (7)

• Figure 1: Overview of the research methods.
• Figure 2: Automated patching with search/replace blocks generated by LLMs
• Figure 3: Distribution of $pss$ scores for LLM-generated solutions, aggregated across all tasks. The left plot groups results by prompting strategy, while the right plot groups results by model. The red dashed line indicates a $pss$ score of 1.0 (i.e., performance equal to the original code). The blue dashed line indicates the median $pss$ score of the developer's solutions across all tasks.
• Figure 4: Heatmap of Vargha-Delaney A12 effect size from pairwise comparisons of all configurations (as well as the original baseline and the developer-written solutions). A cell indicates the value of the configuration for the row versus the configuration in that column. Effect sizes are shown only for pairs with a statistically significant difference ($p < 0.05$ from Wilcoxon signed-rank test).
• Figure 5: A representative example of a strategy match, alignment, and divergence.