BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models

TL;DR

BinMetric addresses the lack of standardized benchmarks for evaluating large language models on binary analysis by introducing a multi-task, real-world benchmark built from 20 open-source projects and 1,000 questions across six tasks. It standardizes data collection, preprocessing, and evaluation via four evaluators, and provides a comparative study of 12 LLMs (open- and closed-source) to reveal strengths and limitations across tasks, with GPT-4 leading overall and open-source CodeLlama-34B showing strong potential. The findings highlight that while LLMs can perform complex analyses, precise binary lifting and assembly synthesis remain challenging, and prompt design along with model size significantly impact performance and efficiency. The work lays groundwork for binary-specific LLM development and a public leaderboard to drive progress in software security applications.

Abstract

Binary analysis remains pivotal in software security, offering insights into compiled programs without source code access. As large language models (LLMs) continue to excel in diverse language understanding and generation tasks, their potential in decoding complex binary data structures becomes evident. However, the lack of standardized benchmarks in this domain limits the assessment and comparison of LLM's capabilities in binary analysis and hinders the progress of research and practical applications. To bridge this gap, we introduce BinMetric, a comprehensive benchmark designed specifically to evaluate the performance of large language models on binary analysis tasks. BinMetric comprises 1,000 questions derived from 20 real-world open-source projects across 6 practical binary analysis tasks, including decompilation, code summarization, assembly instruction generation, etc., which reflect actual reverse engineering scenarios. Our empirical study on this benchmark investigates the binary analysis capabilities of various state-of-the-art LLMs, revealing their strengths and limitations in this field. The findings indicate that while LLMs show strong potential, challenges still exist, particularly in the areas of precise binary lifting and assembly synthesis. In summary, BinMetric makes a significant step forward in measuring the binary analysis capabilities of LLMs, establishing a new benchmark leaderboard, and our study provides valuable insights for the future development of these LLMs in software security.

Figures (5)

• Figure 1: Binary analysis process in case story in §\ref{['sec:case']}.
• Figure 2: Overview framework of BinMetric benchmark.
• Figure 3: Performance comparison between LLMs.
• Figure 4: Results of practical significance (Cohen's d).
• Figure 5: Impact of prompt word and code length on performance