An Empirical Study on the Effectiveness of Large Language Models for Binary Code Understanding

TL;DR

Binary code understanding is challenged by stripped binaries lacking semantic labels. The authors design an automated benchmark and comprehensively evaluate a broad set of LLMs and DL-based models on function name recovery and binary code summarization across multiple architectures and optimizations, including fine-tuned variants and real-world virus case studies. They show that code-domain LLMs (notably CodeLlama and WizardCoder) and general-domain LLMs (notably ChatGPT) can achieve meaningful results, with ChatGPT excelling in summarization while CodeLlama-34b leads in function name recovery, and that fine-tuning yields substantial gains. The work highlights the potential of LLMs to augment reverse engineering workflows, identifies key factors that affect performance (prompt design, pseudo code length, symbol information), and points to future directions like domain-specific models, longer-context architectures, and multi-modal data integration for robust binary analysis.

Abstract

Binary code analysis plays a pivotal role in the field of software security and is widely used in tasks such as software maintenance, malware detection, software vulnerability discovery, patch analysis, etc. However, unlike source code, reverse engineers face significant challenges in understanding binary code due to the lack of intuitive semantic information. Although traditional reverse tools can convert binary code into C-like pseudo code, the lack of code comments and symbolic information such as function names still makes code understanding difficult. In recent years, two groups of techniques have shown promising prospects: (1) Deep learning-based techniques have demonstrated competitive results in tasks related to binary code understanding, furthermore, (2) Large Language Models (LLMs) have been extensively pre-trained at the source-code level for tasks such as code understanding and generation. This has left participants wondering about the capabilities of LLMs in binary code understanding. To this end, this work proposes a benchmark to evaluate the effectiveness of LLMs in real-world reverse engineering scenarios, which covers two key binary code understanding tasks, i.e., function name recovery and binary code summarization. To more comprehensively evaluate, we include binaries with multiple target architectures as well as different optimization options. We gain valuable insights into the capabilities and limitations through extensive empirical studies of popular LLMs using our benchmark. Our evaluations reveal that existing LLMs can understand binary code to a certain extent, thereby improving the efficiency of binary code analysis. Our results highlight the great potential of the LLMs in advancing the field of binary code understanding, and provide new directions for binary code analysis techniques.

Figures (15)

• Figure 1: Application background of binary code understanding.
• Figure 2: Example of source, assembly, stripped assembly and stripped pseudo code snippet.
• Figure 3: An overview of the benchmark dataset construction process.
• Figure 4: Statistics of our benchmark dataset.
• Figure 5: An overview of the evaluation process.