Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis

Michael J. Bommarito

TL;DR

This work tackles the tokenization bottleneck in binary analysis by introducing Binary BPE, a cross-platform family of byte-level BPE tokenizers trained on a large, diverse corpus of binaries. The authors design a nested vocabulary strategy across 4K–64K sizes, apply entropy-filtered, chunk-based BPE on raw bytes, and demonstrate that tokens capture interpretable structures such as file-format headers and ISA patterns, yielding two-to-threefold improvements in context efficiency over raw bytes. They provide extensive ablations on vocabulary scaling, show consistent cross-platform compression gains, and release the tokenizers publicly to enable broad adoption in transformers and binary-focused tooling. The practical impact spans edge-to-datacenter deployments, enabling more efficient malware analysis, reverse engineering workflows, and program understanding through a scalable, open foundation for binary-focused language models and tools.

Abstract

Sequence models for binary analysis are bottlenecked by byte-level tokenization: raw bytes waste precious context window capacity for transformers and other neural network architectures, and many existing text-oriented tokenizers fail on arbitrary 0x00--0xFF sequences. To address this issue, we introduce the Binary BPE tokenizer family, a set of cross-platform Byte Pair Encoding (BPE) tokenizers for executables trained on a large corpus of binaries spanning multiple platforms, architectures, and operating systems, including Linux, Windows, macOS, Android, and malware sources. We release trained tokenizers with vocabularies of 4K, 8K, 16K, 32K, and 64K tokens, enabling both systematic scaling studies and practical deployment from resource-constrained edge devices to high-throughput datacenters. These tokenizers discover interpretable patterns (ELF/PE headers, instruction sequences, cross-platform strings) while yielding multi-byte compression per token. On representative uncompressed executables (e.g., ELF/PE/Mach-O rather than compressed APKs), the Binary BPE tokenizers typically allow for roughly 2-3x more binary content per fixed-length transformer context window than raw bytes, enabling more efficient research and practical deployment for content identification, malware detection, reverse engineering, and optimization. We release the trained Binary BPE tokenizers on HuggingFace, providing a drop-in, open-source foundation for binary-focused language models and context-efficient agentic tools.

Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis

TL;DR

This work tackles the tokenization bottleneck in binary analysis by introducing Binary BPE, a cross-platform family of byte-level BPE tokenizers trained on a large, diverse corpus of binaries. The authors design a nested vocabulary strategy across 4K–64K sizes, apply entropy-filtered, chunk-based BPE on raw bytes, and demonstrate that tokens capture interpretable structures such as file-format headers and ISA patterns, yielding two-to-threefold improvements in context efficiency over raw bytes. They provide extensive ablations on vocabulary scaling, show consistent cross-platform compression gains, and release the tokenizers publicly to enable broad adoption in transformers and binary-focused tooling. The practical impact spans edge-to-datacenter deployments, enabling more efficient malware analysis, reverse engineering workflows, and program understanding through a scalable, open foundation for binary-focused language models and tools.

Abstract

Sequence models for binary analysis are bottlenecked by byte-level tokenization: raw bytes waste precious context window capacity for transformers and other neural network architectures, and many existing text-oriented tokenizers fail on arbitrary 0x00--0xFF sequences. To address this issue, we introduce the Binary BPE tokenizer family, a set of cross-platform Byte Pair Encoding (BPE) tokenizers for executables trained on a large corpus of binaries spanning multiple platforms, architectures, and operating systems, including Linux, Windows, macOS, Android, and malware sources. We release trained tokenizers with vocabularies of 4K, 8K, 16K, 32K, and 64K tokens, enabling both systematic scaling studies and practical deployment from resource-constrained edge devices to high-throughput datacenters. These tokenizers discover interpretable patterns (ELF/PE headers, instruction sequences, cross-platform strings) while yielding multi-byte compression per token. On representative uncompressed executables (e.g., ELF/PE/Mach-O rather than compressed APKs), the Binary BPE tokenizers typically allow for roughly 2-3x more binary content per fixed-length transformer context window than raw bytes, enabling more efficient research and practical deployment for content identification, malware detection, reverse engineering, and optimization. We release the trained Binary BPE tokenizers on HuggingFace, providing a drop-in, open-source foundation for binary-focused language models and context-efficient agentic tools.

Paper Structure

This paper contains 29 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tokenization in Natural Language Processing
  4. Machine Learning for Binary Analysis
  5. Tokenization for Binary Data
  6. Gap and Contribution
  7. Methodology
  8. Training Data Selection
  9. BPE Training Process
  10. Family training setup.
  11. Evaluation Metrics
  12. Results
  13. Vocabulary Analysis
  14. Learned Patterns and Interpretation
  15. Compression Efficiency
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Bytes per token vs. vocabulary size for the Binary BPE family on the 25-file evaluation suite (24 binaries from the stratified samples/ benchmark plus /usr/bin/ls), showing overall (thick line) and per-OS averages. Compression improves steadily with vocabulary size across all platforms.
  • Figure 2: Token-length distributions (stacked) across vocabulary sizes.
  • Figure 3: Coverage of 2-byte sequences vs. vocabulary size.