Magellan: Autonomous Discovery of Novel Compiler Optimization Heuristics with AlphaEvolve

TL;DR

The paper tackles the difficulty of manually engineered compiler optimization heuristics in the face of hardware and software heterogeneity. Magellan combines an LLM-powered policy-proposal component with evolutionary search and autotuning to synthesize compact, deployable C++ compiler passes that integrate with LLVM and similar toolchains. Across case studies on function inlining and register allocation, Magellan discovers policies that match or surpass expert baselines, achieving size reductions of up to $5.23\%$ and end-to-end performance gains, with preliminary XLA results indicating portability. The work demonstrates substantial productivity gains and reduced engineering effort by turning heuristic design into reusable, concise C++ policies that preserve correctness and deployability across evolving compiler stacks.

Abstract

Modern compilers rely on hand-crafted heuristics to guide optimization passes. These human-designed rules often struggle to adapt to the complexity of modern software and hardware and lead to high maintenance burden. To address this challenge, we present Magellan, an agentic framework that evolves the compiler pass itself by synthesizing executable C++ decision logic. Magellan couples an LLM coding agent with evolutionary search and autotuning in a closed loop of generation, evaluation on user-provided macro-benchmarks, and refinement, producing compact heuristics that integrate directly into existing compilers. Across several production optimization tasks, Magellan discovers policies that match or surpass expert baselines. In LLVM function inlining, Magellan synthesizes new heuristics that outperform decades of manual engineering for both binary-size reduction and end-to-end performance. In register allocation, it learns a concise priority rule for live-range processing that matches intricate human-designed policies on a large-scale workload. We also report preliminary results on XLA problems, demonstrating portability beyond LLVM with reduced engineering effort.

Figures (6)

• Figure 1: Overview of Magellan. We use LLVM as the demonstration, but Magellan can be readily applied to other compilers.
• Figure 2: Evolvable blocks for function inlining.
• Figure 3: Evolutionary curve of binary size reduction on internal search benchmark. Each program sample corresponds to a specific policy template combined with a set of hyperparameters proposed by the autotuner. The initial points differ between the two methods because predefined features influence the inlining decisions prior to search.
• Figure 4: Temporal generalization for function inlining.
• Figure 5: Domain generalization for function inlining.