On the Role of Pre-trained Embeddings in Binary Code Analysis

TL;DR

This work critically evaluates the role of pre-trained embeddings in binary code analysis by benchmarking multiple instruction and function embeddings on 5 downstream tasks using a large Debian-derived corpus. It shows that with ample labeled data, pre-trained embeddings yield little advantage over end-to-end learning, and even random embeddings can perform competitively; end-to-end models generally provide the best average performance, while PalmTree shines when labeled data is scarce. The authors provide practical guidelines: start with end-to-end learning, resort to PalmTree or other pre-trained embeddings only under data or compute constraints, and publish embeddings to enable reproducibility. The study challenges the assumption that pre-training transfers benefits from NLP to binary code analysis and underscores the importance of task- and data-aware modeling choices.

Abstract

Deep learning has enabled remarkable progress in binary code analysis. In particular, pre-trained embeddings of assembly code have become a gold standard for solving analysis tasks, such as measuring code similarity or recognizing functions. These embeddings are capable of learning a vector representation from unlabeled code. In contrast to natural language processing, however, label information is not scarce for many tasks in binary code analysis. For example, labeled training data for function boundaries, optimization levels, and argument types can be easily derived from debug information provided by a compiler. Consequently, the main motivation of embeddings does not transfer directly to binary code analysis. In this paper, we explore the role of pre-trained embeddings from a critical perspective. To this end, we systematically evaluate recent embeddings for assembly code on five downstream tasks using a corpus of 1.2 million functions from the Debian distribution. We observe that several embeddings perform similarly when sufficient labeled data is available, and that differences reported in prior work are hardly noticeable. Surprisingly, we find that end-to-end learning without pre-training performs best on average, which calls into question the need for specialized embeddings. By varying the amount of labeled data, we eventually derive guidelines for when embeddings offer advantages and when end-to-end learning is preferable for binary code analysis.

Figures (14)

• Figure 1: Schematic comparison of pre-training task with downstream task and conventional end-to-end learning.
• Figure 2: Example of x86-64 assembly instructions.
• Figure 3: Dataset creation pipeline. A function corpus is build from all available training packages. Based on this corpus the embeddings are trained. Using the pre-trained embeddings, each Debian package is transformed into a dataset file containing the embedded functions and corresponding labels.
• Figure 4: General accuracy of the considered embeddings in relation to the available labeled training data (shards)
• Figure 5: General accuracy of the considered embeddings in relation to the available labeled training data (shards).