Should AI Optimize Your Code? A Comparative Study of Classical Optimizing Compilers Versus Current Large Language Models

TL;DR

The paper benchmarks AI-driven code optimization against classical optimizing compilers using a new PCAOT evaluation framework and the PCB benchmark. It analyzes three prompting strategies and two LLMs (CodeLlama-70B, GPT-4) compared to CETUS, PLUTO, and ROSE, revealing that LLMs can yield speedups on small, section-level code but struggle with larger programs and many optimization patterns. DIP prompts and explicit compiler strategies significantly boost LLM performance, yet only CodeLlama-70B achieves notable gains (up to ~1.95x), while PLUTO and CETUS still often outperform LLMs on broader suites. The findings indicate LLMs are not yet ready to replace traditional compilers for automatic optimization, though they show potential with further advances and rigorous verification. The work provides a rigorous, automatable framework for correctness and performance validation and introduces PCB as a challenging, unbiased benchmark for future AOT development.

Abstract

Traditional optimizing compilers have played an important role in adapting to the growing complexity of modern software systems. The need for efficient parallel programming in current architectures requires strong optimization techniques. The beginning of Large Language Models (LLMs) raises intriguing questions about the potential of these AI approaches to revolutionize code optimization methodologies. This work aims to answer an essential question for the compiler community: "Can AI-driven models revolutionize the way we approach code optimization?". To address this question, we present a comparative analysis between three classical optimizing compilers and two recent large language models, evaluating their respective abilities and limitations in optimizing code for maximum efficiency. In addition, we introduce a benchmark suite of challenging optimization patterns and an automatic mechanism for evaluating the performance and correctness of the code generated by LLMs. We used three different prompting strategies to evaluate the performance of the LLMs, Simple Instruction (IP), Detailed Instruction Prompting (DIP), and Chain of Thought (CoT). A key finding is that while LLMs have the potential to outperform current optimizing compilers, they often generate incorrect code on large code sizes, calling for automated verification methods. In addition, expressing a compiler strategy as part of the LLMs prompt substantially improves its overall performance. Our evaluation across three benchmark suites shows CodeLlama-70B as the superior LLM, capable of achieving speedups of up to x1.75. Additionally, CETUS is the best among the current optimizing compilers, achieving a maximum speedup of 1.67x. We also found substantial differences among the three prompting strategies.

Figures (7)

• Figure 1: Architecture of the PCAOT Environment.
• Figure 2: Experimental section instrumentation. Example taken from subroutine PO_Function_Version_1 of the PCB suite.
• Figure 3: Average failure rate of the Large Language Models across benchmarks when generating programs with increasing code sizes.
• Figure 4: Correctness Evaluation for the challenge patterns: Parallelization at the Outermost level (P-Outermost), Parallelization for function calls (P-Function Calls), Array Reduction, Parallel-Regions, NOWAIT, and Dynamic Scheduling (Dynamic Sch).
• Figure 5: Correctness Evaluation for the challenge patterns: Parallelization at the Outermost level and Parallelization for function calls in non-parallel Scenarios. The expected behavior in this case is that the LLM recognizes the code as non parallelizable.