SBAN: A Framework & Multi-Dimensional Dataset for Large Language Model Pre-Training and Software Code Mining

TL;DR

SBAN addresses the need for a unified, multi-modal dataset that spans binary, assembly, source code, and natural language descriptions to enhance large language model pre-training for software code mining. By integrating four interconnected representations and providing a complete pipeline from data collection to validation, SBAN enables cross-modal learning for tasks such as malware detection, code understanding, and program analysis. The paper details dataset architecture, decompilation-based generation of source-like code, NLP preprocessing, and rigorous data integrity checks, followed by an unsupervised pre-training experiment showing improved perplexity on benign C++ code. Its contributions offer a robust foundation for scalable LLM training in software mining, with practical impact on cybersecurity, code intelligence, and multimodal learning, while acknowledging limitations and the need for diversification and responsible deployment.

Abstract

This paper introduces SBAN (Source code, Binary, Assembly, and Natural Language Description), a large-scale, multi-dimensional dataset designed to advance the pre-training and evaluation of large language models (LLMs) for software code analysis. SBAN comprises more than 3 million samples, including 2.9 million benign and 672,000 malware respectively, each represented across four complementary layers: binary code, assembly instructions, natural language descriptions, and source code. This unique multimodal structure enables research on cross-representation learning, semantic understanding of software, and automated malware detection. Beyond security applications, SBAN supports broader tasks such as code translation, code explanation, and other software mining tasks involving heterogeneous data. It is particularly suited for scalable training of deep models, including transformers and other LLM architectures. By bridging low-level machine representations and high-level human semantics, SBAN provides a robust foundation for building intelligent systems that reason about code. We believe that this dataset opens new opportunities for mining software behavior, improving security analytics, and enhancing LLM capabilities in pre-training and fine-tuning tasks for software code mining.

Figures (5)

• Figure 1: Relation between different sources in the SBAN dataset.
• Figure 2: A view of SBAN dataset pipline.
• Figure 3: A view of a single sample from the SBAN.
• Figure 4: Framework for evaluating Decompiled Files
• Figure 5: Distribution of Perplexity Scores for both Models on Benign Samples