Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in Lifted Compiled Code

TL;DR

This work tackles vulnerability detection in lifted binaries by learning code semantics from LLVM IR and benchmarking word2vec against bidirectional transformers (BERT, RoBERTa) in conjunction with LSTM classifiers. Using about 48k LLVM functions from the Juliet dataset, the study finds that word2vec Skip-Gram achieves the highest validation accuracy (≈92%), outperforming BERT and RoBERTa under a small-data regime. The results challenge the assumption that complex contextual embeddings always outperform simpler word-level models when data is limited, and they highlight the practical considerations of memory and max-length constraints. Overall, the paper provides guidance on selecting embedding techniques for compiler-independent semantic representations in machine learning-based vulnerability detection for compiled binaries.

Abstract

Detecting vulnerabilities within compiled binaries is challenging due to lost high-level code structures and other factors such as architectural dependencies, compilers, and optimization options. To address these obstacles, this research explores vulnerability detection using natural language processing (NLP) embedding techniques with word2vec, BERT, and RoBERTa to learn semantics from intermediate representation (LLVM IR) code. Long short-term memory (LSTM) neural networks were trained on embeddings from encoders created using approximately 48k LLVM functions from the Juliet dataset. This study is pioneering in its comparison of word2vec models with multiple bidirectional transformers (BERT, RoBERTa) embeddings built using LLVM code to train neural networks to detect vulnerabilities in compiled binaries. Word2vec Skip-Gram models achieved 92% validation accuracy in detecting vulnerabilities, outperforming word2vec Continuous Bag of Words (CBOW), BERT, and RoBERTa. This suggests that complex contextual embeddings may not provide advantages over simpler word2vec models for this task when a limited number (e.g. 48K) of data samples are used to train the bidirectional transformer-based models. The comparative results provide novel insights into selecting optimal embeddings for learning compiler-independent semantic code representations to advance machine learning detection of vulnerabilities in compiled binaries.

Figures (2)

• Figure 1: Building Embedding Models Using Lifted Code
• Figure 2: Training Neural Networks to Detect Buffer Overflows