ShieldedCode: Learning Robust Representations for Virtual Machine Protected Code

TL;DR

ShieldedCode presents a protection-aware learning framework to robustly represent VM-protected code, addressing reverse-engineering threats where traditional VMP relies on rigid transformations. By constructing large paired datasets, introducing a three-level hierarchical dependency model, and jointly optimizing language modeling with functionality- and protection-aware contrastive losses plus a protection-effectiveness objective, the approach enables generation, comparison, and reasoning over protected code. Two-stage continual pre-training and fine-tuning, together with a protection-strength ranking via Protection Effectiveness Optimization, yield strong gains in VM code generation and binary code similarity tasks, and demonstrate improved resistance to reverse-engineering attacks. The work establishes a new direction in learning-based software defense, offering practical improvements in robustness and a framework adaptable to diverse protection schemes.

Abstract

Large language models (LLMs) have achieved remarkable progress in code generation, yet their potential for software protection remains largely untapped. Reverse engineering continues to threaten software security, while traditional virtual machine protection (VMP) relies on rigid, rule-based transformations that are costly to design and vulnerable to automated analysis. In this work, we present the first protection-aware framework that learns robust representations of VMP-protected code. Our approach builds large-scale paired datasets of source code and normalized VM implementations, and introduces hierarchical dependency modeling at intra-, preceding-, and inter-instruction levels. We jointly optimize language modeling with functionality-aware and protection-aware contrastive objectives to capture both semantic equivalence and protection strength. To further assess resilience, we propose a protection effectiveness optimization task that quantifies and ranks different VM variants derived from the same source. Coupled with a two-stage continual pre-training and fine-tuning pipeline, our method enables models to generate, compare, and reason over protected code. Extensive experiments show that our framework significantly improves robustness across diverse protection levels, opening a new research direction for learning-based software defense. In this work, we present ShieldedCode, the first protection-aware framework that learns robust representations of VMP-protected code. Our method achieves 26.95% Pass@1 on L0 VM code generation compared to 22.58% for GPT-4o., and improves binary similarity detection Recall@1 by 10% over state of art methods like jTrans.

Figures (5)

• Figure 1: Overview of our key components, including hierarchical dependencies in polymorphic execution, as well as the PCL and FCL objectives. The upper part illustrates the hierarchical attention mask used in polymorphic execution. Hierarchical attention mask showing the hierarchical instruction-aware attention pattern. The matrix visualizes how tokens attend to each other across different instruction boundaries. Pink regions represent intra-instruction causal attention within each instruction block. Light blue regions show preceding-instruction attention where tokens can access previous [INST-X] boundaries. Dark blue regions indicate inter-instruction attention enabling communication between [INST-X] tokens. Purple regions represent positions with no attention. The lower part presents the objectives of FCL and PCL.
• Figure 2: Complete transformation pipeline from C source code to VMP-protected assembly. The source code implements a min-max finding algorithm. The O2 assembly shows compiler optimization with complex control flow. The VMP version uses virtual instructions with [VINST-X] markers and virtual registers (prefixed with 'v'), demonstrating how our hierarchical attention mechanism processes instruction boundaries and dependencies.
• Figure 3: Heatmap visualization of Recall@1 performance across different values of $K$ (50, 100, 200, 500). Each subfigure illustrates method-wise results on six benchmarks, with lighter colors indicating higher values. The fifth subfigure summarizes the averaged performance across all benchmarks for each method under varying $K$. This unified view highlights the relative strengths of different approaches across both individual datasets and performance.
• Figure 4: t-SNE visualization of function embeddings across optimization-protection levels. O1+L1 shows higher dispersion explaining lower performance, while O0+L1 forms tight clusters indicating better learned representations.
• Figure 5: Complete transformation pipeline from C source code to VMP-protected assembly. The source code implements a min-max finding algorithm. The O2 assembly shows compiler optimization with complex control flow. The VMP version uses virtual instructions with [VINST-X] markers and virtual registers (prefixed with 'v'), demonstrating how our hierarchical attention mechanism processes instruction boundaries and dependencies.

Theorems & Definitions (5)

• Theorem A.1: Hierarchical Mask Expressivity
• Theorem A.2: Functionality and Protection Contrastive Alignment
• Corollary 1: Weighted Alignment Decay
• Theorem A.3: PEO Top-1 Guarantee with Hard Negative Mining
• Proposition A.4: Structured Alignment under Joint Training