Synthetic Datasets for Program Similarity Research

TL;DR

High-quality datasets for binary program similarity are scarce and labels often fail to capture true semantic similarity. The authors introduce HELIX, a language-agnostic framework that generates large synthetic datasets by recombining labeled program components, and Blind HELIX, a component-extraction tool based on program slicing. Ground-truth similarity is defined by the overlap of component-label sets, enabling rigorous evaluation against multiple notions of similarity and across several tools (ssdeep, sdhash, TLSH, LZJD, BinDiff). The approach yields effectively infinite dataset size with quick generation times (e.g., 2:29 for 256 samples) and is open-sourced to support reproducible benchmarking in program similarity research.

Abstract

Program similarity has become an increasingly popular area of research with various security applications such as plagiarism detection, author identification, and malware analysis. However, program similarity research faces a few unique dataset quality problems in evaluating the effectiveness of novel approaches. First, few high-quality datasets for binary program similarity exist and are widely used in this domain. Second, there are potentially many different, disparate definitions of what makes one program similar to another and in many cases there is often a large semantic gap between the labels provided by a dataset and any useful notion of behavioral or semantic similarity. In this paper, we present HELIX - a framework for generating large, synthetic program similarity datasets. We also introduce Blind HELIX, a tool built on top of HELIX for extracting HELIX components from library code automatically using program slicing. We evaluate HELIX and Blind HELIX by comparing the performance of program similarity tools on a HELIX dataset to a hand-crafted dataset built from multiple, disparate notions of program similarity. Using Blind HELIX, we show that HELIX can generate realistic and useful datasets of virtually infinite size for program similarity research with ground truth labels that embody practical notions of program similarity. Finally, we discuss the results and reason about relative tool ranking.

Figures (6)

• Figure 1: The design of HELIX - sets of components and transforms are combined using a blueprint to generate samples in a synthetic dataset.
• Figure 2: Blind HELIX's use of program slicing - a slice is taken starting at each exported function in the library to generate a single HELIX component.
• Figure 3: Distribution of ground truth similarity scores in the HELIX-generated dataset. The dataset was generated such that there would not be an unreasonable abundance of low-similarity or high-similarity scores to avoid benefiting biased similarity metrics.
• Figure 4: Program similarity metric performance on the Blind HELIX and manually-labeled datasets - lower is better.
• Figure 5: Comparative program similarity metric performance on manually-labeled dataset similarity categories - lower is better. The "naive" metric represents always guessing a similarity of 0.5.