CLAP: Learning Transferable Binary Code Representations with Natural Language Supervision

TL;DR

CLAP tackles the limited transferability of binary code representations by introducing natural language supervision to align assembly code with semantic explanations. The method combines a scalable dataset engine that produces 195 million binary-explanation pairs with a two-stage CLAP engine that pre-trains an assembly encoder and then performs contrastive learning against a text encoder, enabling strong zero-shot and few-shot transfer across BCSD, crypto identification, and protocol categorization tasks. Empirical results show CLAP surpasses state-of-the-art baselines in zero-shot settings and remains highly competitive with minimal task-specific training, highlighting the practical potential for semantic-rich binary analysis. The work provides insights into the role of language supervision in bridging code and natural-language semantics, and offers a usable resource (code and model) to accelerate future research in binary analysis and security applications.

Abstract

Binary code representation learning has shown significant performance in binary analysis tasks. But existing solutions often have poor transferability, particularly in few-shot and zero-shot scenarios where few or no training samples are available for the tasks. To address this problem, we present CLAP (Contrastive Language-Assembly Pre-training), which employs natural language supervision to learn better representations of binary code (i.e., assembly code) and get better transferability. At the core, our approach boosts superior transfer learning capabilities by effectively aligning binary code with their semantics explanations (in natural language), resulting a model able to generate better embeddings for binary code. To enable this alignment training, we then propose an efficient dataset engine that could automatically generate a large and diverse dataset comprising of binary code and corresponding natural language explanations. We have generated 195 million pairs of binary code and explanations and trained a prototype of CLAP. The evaluations of CLAP across various downstream tasks in binary analysis all demonstrate exceptional performance. Notably, without any task-specific training, CLAP is often competitive with a fully supervised baseline, showing excellent transferability. We release our pre-trained model and code at https://github.com/Hustcw/CLAP.

Figures (11)

• Figure 1: A real-world function named xstrmode (a) and its assembly code (b)
• Figure 2: Overview of our primary workflow, consisting of two main components, the dataset engine and the CLAP engine. The dataset engine compiles source code from the Ubuntu Repository into assembly code and uses GPT to generate explanations, forming an explanation dataset. We then fine-tune LLaMA model using the source code with the corresponding explanation, resulting in a shadow model that generates the augmented explanation dataset. Within the CLAP engine, we first pre-train the assembly code dataset to develop an initial assembly encoder. By employing contrastive learning with this assembly encoder and a text encoder on both assembly code and augmented explanation dataset, we produce the final CLAP model.
• Figure 3: Illustration of CLAP-ASM. The raw assembly code is first rebased and tokenized. Then, each token is converted to a token embedding, a position embedding, and an instruction embedding, and the sum of these three is output as the embedding. The instruction embedding remembers instruction boundaries and works with jump symbols to comprehend control flow. The token embedding of the jump symbol (e.g., $\mathbf{E_\mathtt{INSTR13}}$) shares parameters with the target instruction embedding (e.g., $\mathbf{I_{\mathtt{13}}})$.
• Figure 4: The comparison between contrastive pre-training from scratch and contrastive pre-training based on the first pre-training stage model (batch size = 1024, epoch <= 1). The left figure shows the InfoNCE loss during pre-training. The right figure shows the Recall@1 result in the validation dataset, in which the model needs to select the sole natural language explanation that matches the assembly code from 65,536 natural language explanations.
• Figure 5: Zero-Shot Inference of CLAP