BinPool: A Dataset of Vulnerabilities for Binary Security Analysis

TL;DR

This work tackles the scarcity of large, real-world binary vulnerability datasets by introducing BinPool, a public, automated corpus derived from historical Debian vulnerabilities. The authors map CVEs to CWEs, retrieve affected and fixed package versions from NVD and Debian Snapshots, and reconstruct paired vulnerable and patched binaries across four optimization levels, accompanied by detailed source- and binary-level metadata. The dataset comprises 603 CVEs across 89 CWEs, 162 Debian packages, and 6144 binaries (910 source functions and 7280 binary functions), enabling evaluation of vulnerability discovery and binary similarity methods, as well as benchmarking with static analysis tools like angr. They provide open-source tooling and a GitHub repository, with plans to grow coverage and enrich inter-procedural data flows, thereby offering a practical, scalable resource for security analysis research.

Abstract

The development of machine learning techniques for discovering software vulnerabilities relies fundamentally on the availability of appropriate datasets. The ideal dataset consists of a large and diverse collection of real-world vulnerabilities, paired so as to contain both vulnerable and patched versions of each program. Naturally, collecting such datasets is a laborious and time-consuming task. Within the specific domain of vulnerability discovery in binary code, previous datasets are either publicly unavailable, lack semantic diversity, involve artificially introduced vulnerabilities, or were collected using static analyzers, thereby themselves containing incorrectly labeled example programs. In this paper, we describe a new publicly available dataset which we dubbed Binpool, containing numerous samples of vulnerable versions of Debian packages across the years. The dataset was automatically curated, and contains both vulnerable and patched versions of each program, compiled at four different optimization levels. Overall, the dataset covers 603 distinct CVEs across 89 CWE classes, 162 Debian packages, and contains 6144 binaries. We argue that this dataset is suitable for evaluating a range of security analysis tools, including for vulnerability discovery, binary function similarity, and plagiarism detection.

Figures (2)

• Figure 1: Construction process of the BinPool dataset. (a) In the first phase, we collect data about the vulnerabilities by gathering CVE-IDs, CWEs, and the affected and fixed package versions from the Debian snapshots and NVD databases. (b) In the second phase, we build packages for both vulnerable and patched versions. (c) In the last phase, we extract detailed metadata, including function names and vulnerability locations (both at source and binary levels).
• Figure 2: Structure of the BinPool dataset. The central CSV file contains information about CVE and CWE-IDs, version numbers, and links to source code. In parallel, the dataset is organized according to the vulnerability IDs. Each vulnerability includes metadata about the function names, module names, and affected code locations in pkl files, and versions of the vulnerable and patched deb files and binaries obtained from different optimization levels.