XuanJia: A Comprehensive Virtualization-Based Code Obfuscator for Binary Protection

TL;DR

XuanJia targets a critical blind spot in virtualization-based obfuscation: the exception-handling (EH) metadata that enables reverse engineering to reveal program structure. It introduces ABI-Compliant EH Shadowing, which redirects EH processing into a protected VM while preserving the OS ABI, thereby eliminating static EH leakage without affecting runtime correctness. The framework translates native code to VM bytecode, uses a direct-threaded interpreter, and employs DSLs for extensibility, achieving end-to-end protection with 385 x86 encodings and 155 VM handlers. Evaluation shows semantic equivalence, strong resistance to static analysis tools like IDA Pro, and modest space overhead with manageable runtime costs, highlighting practical viability for protecting EH-sensitive binaries.

Abstract

Virtualization-based binary obfuscation is widely adopted to protect software intellectual property, yet existing approaches leave exception-handling (EH) metadata unprotected to preserve ABI compatibility. This exposed metadata leaks rich structural information, such as stack layouts, control-flow boundaries, and object lifetimes, which can be exploited to facilitate reverse engineering. In this paper, we present XuanJia, a comprehensive VM-based binary obfuscation framework that provides end-to-end protection for both executable code and exception-handling semantics. At the core of XuanJia is ABI-Compliant EH Shadowing, a novel exception-aware protection mechanism that preserves compatibility with unmodified operating system runtimes while eliminating static EH metadata leakage. XuanJia replaces native EH metadata with ABI-compliant shadow unwind information to satisfy OS-driven unwinding, and securely redirects exception handling into a protected virtual machine where the genuine EH semantics are decrypted, reversed, and replayed using obfuscated code. We implement XuanJia from scratch, supporting 385 x86 instruction encodings and 155 VM handler templates, and design it as an extensible research testbed. We evaluate XuanJia across correctness, resilience, and performance dimensions. Our results show that XuanJia preserves semantic equivalence under extensive dynamic and symbolic testing, effectively disrupts automated reverse-engineering tools such as IDA Pro, and incurs negligible space overhead and modest runtime overhead. These results demonstrate that XuanJia achieves strong protection of exception-handling logic without sacrificing correctness or practicality.

Figures (6)

• Figure 1: Workflow of VM-based obfuscation. The process transforms native instructions through three stages: instruction parsing, instruction translation, and VM integration. The original code is replaced by a VM entry stub that redirects control flow to the VM interpreter for VM bytecode execution. (VIP = virtual instruction pointer; disp. = dispatcher)
• Figure 2: The intrinsic coupling between binary code and EH metadata. The Global Section describes the static stack frame structure via unwind codes, while the Local Section encodes dynamic object lifecycles via LSData. This mechanism renders the metadata a faithful reflection of the original program logic.
• Figure 3: Overview of XuanJia’s exception-aware protection workflow. During static transformation, XuanJia provides comprehensive protection for both executable code and exception-handling metadata. At runtime, protected code executes within XuanJia’s VM interpreter. After a dummy global unwinding step to preserve ABI compatibility, the remaining exception-handling logic is securely processed within the VM interpreter, achieving both compatibility and strong protection.
• Figure 4: The architecture of XuanJia’s static protection engine. The workflow operates in three decoupled stages: Instruction Parsing, Instruction Translation, and VM Integration. XuanJia employs custom domain-specific languages (translation and handler DSL) to decouple obfuscation logic from the underlying architecture. Furthermore, it adopts a pass-driven design with dedicated interfaces (VM and backend pass) to facilitate flexible transformation.
• Figure 5: Overview of the runtime mechanism of ABI-Compliant EH Shadowing in XuanJia. The detailed structure of the EH metadata within the red box corresponds to the layout depicted in Figure \ref{['fig:mirror']}.