Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SuperCoder: Assembly Program Superoptimization with Large Language Models

Anjiang Wei, Tarun Suresh, Huanmi Tan, Yinglun Xu, Gagandeep Singh, Ke Wang, Alex Aiken

TL;DR

This work investigates using large language models to perform superoptimization on assembly code, aiming to surpass gcc -O3 outputs while preserving semantics. It introduces the first large-scale assembly benchmark (8,072 programs) and demonstrates that RL-finetuned models (SuperCoder) can achieve high correctness (up to 95.0%) and significantly improve performance (up to 1.46× average speedup) over the baseline. The study shows that some LLMs can outperform compiler heuristics under test-based validation, with Best-of-N sampling and iterative refinement further boosting results. It also analyzes transformation categories and discusses prompts, reward designs, and limitations, laying groundwork for future research in low-level program performance optimization beyond traditional compilers.

Abstract

Superoptimization is the task of transforming a program into a faster one while preserving its input-output behavior. In this work, we investigate whether large language models (LLMs) can serve as superoptimizers, generating assembly programs that outperform code already optimized by industry-standard compilers. We construct the first large-scale benchmark for this problem, consisting of 8,072 assembly programs averaging 130 lines, in contrast to prior datasets restricted to 2-15 straight-line, loop-free programs. We evaluate 23 LLMs on this benchmark and find that the strongest baseline, Claude-opus-4, achieves a 51.5% test-passing rate and a 1.43x average speedup over gcc -O3. To further enhance performance, we fine-tune models with reinforcement learning, optimizing a reward function that integrates correctness and performance speedup. Starting from Qwen2.5-Coder-7B-Instruct (61.4% correctness, 1.10x speedup), the fine-tuned model SuperCoder attains 95.0% correctness and 1.46x average speedup, with additional improvement enabled by Best-of-N sampling and iterative refinement. Our results demonstrate, for the first time, that LLMs can be applied as superoptimizers for assembly programs, establishing a foundation for future research in program performance optimization beyond compiler heuristics.

SuperCoder: Assembly Program Superoptimization with Large Language Models

TL;DR

This work investigates using large language models to perform superoptimization on assembly code, aiming to surpass gcc -O3 outputs while preserving semantics. It introduces the first large-scale assembly benchmark (8,072 programs) and demonstrates that RL-finetuned models (SuperCoder) can achieve high correctness (up to 95.0%) and significantly improve performance (up to 1.46× average speedup) over the baseline. The study shows that some LLMs can outperform compiler heuristics under test-based validation, with Best-of-N sampling and iterative refinement further boosting results. It also analyzes transformation categories and discusses prompts, reward designs, and limitations, laying groundwork for future research in low-level program performance optimization beyond traditional compilers.

Abstract

Superoptimization is the task of transforming a program into a faster one while preserving its input-output behavior. In this work, we investigate whether large language models (LLMs) can serve as superoptimizers, generating assembly programs that outperform code already optimized by industry-standard compilers. We construct the first large-scale benchmark for this problem, consisting of 8,072 assembly programs averaging 130 lines, in contrast to prior datasets restricted to 2-15 straight-line, loop-free programs. We evaluate 23 LLMs on this benchmark and find that the strongest baseline, Claude-opus-4, achieves a 51.5% test-passing rate and a 1.43x average speedup over gcc -O3. To further enhance performance, we fine-tune models with reinforcement learning, optimizing a reward function that integrates correctness and performance speedup. Starting from Qwen2.5-Coder-7B-Instruct (61.4% correctness, 1.10x speedup), the fine-tuned model SuperCoder attains 95.0% correctness and 1.46x average speedup, with additional improvement enabled by Best-of-N sampling and iterative refinement. Our results demonstrate, for the first time, that LLMs can be applied as superoptimizers for assembly programs, establishing a foundation for future research in program performance optimization beyond compiler heuristics.
Paper Structure (52 sections, 5 equations, 6 figures, 6 tables)

This paper contains 52 sections, 5 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models for Code.
  4. Learning-Based Code Optimization.
  5. Methodology
  6. Task Definition
  7. Performance and Speedup.
  8. Optimization Objective.
  9. Dataset Construction
  10. Reinforcement Learning
  11. Optimization with PPO and GRPO.
  12. Reward Function Design.
  13. Best-of-N Sampling, Supervised Fine-Tuning, and Iterative Refinement
  14. Best-of-N Sampling.
  15. Supervised Fine-Tuning.
  16. ...and 37 more sections

Figures (6)

  • Figure 1: Overview of the assembly code optimization task. Given a C program and its baseline assembly from gcc -O3, an LLM is fine-tuned with PPO or GRPO to generate improved assembly. The reward function reflects correctness and performance based on test execution.
  • Figure 2: Best-of-N sampling results.
  • Figure 3: Iterative refinement results.
  • Figure A1: Case study comparing the C code, baseline assembly produced by gcc -O3, and optimized assembly generated by Claude-Opus-4. The model successfully replaces the loop with the specialized hardware instruction popcnt, resulting in a significantly more concise implementation.
  • Figure A2: Case study comparing the baseline assembly code snippet produced by gcc -O3 and the optimized assembly code snippet generated by Claude-Opus-4. Claude-Opus-4 eliminates the entry-path unconditional jump and alignment padding by fusing GCC’s two-block loop into a single, simpler control-flow structure, calls printf@PLT directly (a simpler function variant without security checks), and removes gcc’s stack-protector canary check.
  • ...and 1 more figures