Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EXHIB: A Benchmark for Realistic and Diverse Evaluation of Function Similarity in the Wild

Yiming Fan, Jun Yeon Won, Ding Zhu, Melih Sirlanci, Mahdi Khalili, Carter Yagemann

Abstract

Binary Function Similarity Detection (BFSD) is a core problem in software security, supporting tasks such as vulnerability analysis, malware classification, and patch provenance. In the past few decades, numerous models and tools have been developed for this application; however, due to the lack of a comprehensive universal benchmark in this field, researchers have struggled to compare different models effectively. Existing datasets are limited in scope, often focusing on a narrow set of transformations or types of binaries, and fail to reflect the full diversity of real-world applications. We introduce EXHIB, a benchmark comprising five realistic datasets collected from the wild, each highlighting a distinct aspect of the BFSD problem space. We evaluate 9 representative models spanning multiple BFSD paradigms on EXHIB and observe performance degradations of up to 30% on firmware and semantic datasets compared to standard settings, revealing substantial generalization gaps. Our results show that robustness to low- and mid-level binary variations does not generalize to high-level semantic differences, underscoring a critical blind spot in current BFSD evaluation practices.

EXHIB: A Benchmark for Realistic and Diverse Evaluation of Function Similarity in the Wild

Abstract

Binary Function Similarity Detection (BFSD) is a core problem in software security, supporting tasks such as vulnerability analysis, malware classification, and patch provenance. In the past few decades, numerous models and tools have been developed for this application; however, due to the lack of a comprehensive universal benchmark in this field, researchers have struggled to compare different models effectively. Existing datasets are limited in scope, often focusing on a narrow set of transformations or types of binaries, and fail to reflect the full diversity of real-world applications. We introduce EXHIB, a benchmark comprising five realistic datasets collected from the wild, each highlighting a distinct aspect of the BFSD problem space. We evaluate 9 representative models spanning multiple BFSD paradigms on EXHIB and observe performance degradations of up to 30% on firmware and semantic datasets compared to standard settings, revealing substantial generalization gaps. Our results show that robustness to low- and mid-level binary variations does not generalize to high-level semantic differences, underscoring a critical blind spot in current BFSD evaluation practices.

Paper Structure

This paper contains 55 sections, 3 equations, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Background and Motivation
  4. Problem Description
  5. Binary Analysis.
  6. Binary Function Similarity Comparison.
  7. Methods Systematization
  8. Fuzzy hashing
  9. Graph based methods
  10. NLP based methods
  11. Other notable methods
  12. Motivation
  13. Dataset Design
  14. The Standard Dataset
  15. The Firmware Dataset
  16. ...and 40 more sections

Figures (7)

  • Figure 1: Definition of the BFSD Problem
  • Figure 2: Systematized Evolution Graph of the Three Main Trends of the BFSD Problem Solutions
  • Figure 3: Comparison of the recall at different K values for the Standard Dataset.
  • Figure 4: Recall graphs of Obfuscation and Semantic Dataset
  • Figure 5: A Comparison of the Similarity Score Distributions.
  • ...and 2 more figures