A Large Scale Study of AI-based Binary Function Similarity Detection Techniques for Security Researchers and Practitioners

TL;DR

This work addresses the need for scalable, realistic evaluation of AI-based Binary Function Similarity Detection (BFSD) by constructing two large datasets, BinAtlas and BinAres, and benchmarking nine tools. It reveals mode-specific strengths and weaknesses, demonstrates that combining tools from different representations yields significant gains (up to $13.4\%$ in $MRR$ and improved F1 on vulnerability detection), and provides an actionable strategy for practical BFSD deployment in real-world security tasks. The study also highlights limitations related to function inlining, cross-architecture variability, and the gap between function-level similarity and vulnerability localization, while releasing open-source resources to catalyze future research. Overall, the results advance the applicability of BFSD in large-scale security workflows and point to architecture-agnostic, higher-level representations as a promising direction for future work.

Abstract

Binary Function Similarity Detection (BFSD) is a foundational technique in software security, underpinning a wide range of applications including vulnerability detection, malware analysis. Recent advances in AI-based BFSD tools have led to significant performance improvements. However, existing evaluations of these tools suffer from three key limitations: a lack of in-depth analysis of performance-influencing factors, an absence of realistic application analysis, and reliance on small-scale or low-quality datasets. In this paper, we present the first large-scale empirical study of AI-based BFSD tools to address these gaps. We construct two high-quality and diverse datasets: BinAtlas, comprising 12,453 binaries and over 7 million functions for capability evaluation; and BinAres, containing 12,291 binaries and 54 real-world 1-day vulnerabilities for evaluating vulnerability detection performance in practical IoT firmware settings. Using these datasets, we evaluate nine representative BFSD tools, analyze the challenges and limitations of existing BFSD tools, and investigate the consistency among BFSD tools. We also propose an actionable strategy for combining BFSD tools to enhance overall performance (an improvement of 13.4%). Our study not only advances the practical adoption of BFSD tools but also provides valuable resources and insights to guide future research in scalable and automated binary similarity detection.

Figures (4)

• Figure 1: Workflow of a typical BFSD application.
• Figure 2: Research questions in this study.
• Figure 3: MRR by the size of function pool. (Solid lines: non-inlined dataset; dashed lines: inlined dataset.)
• Figure 4: The overlap coefficient between tool pairs.