Static Detection of Core Structures in Tigress Virtualization-Based Obfuscation Using an LLVM Pass

TL;DR

Malware often uses virtualization-based obfuscation to thwart analysis by translating instructions into VM bytecode, creating large, complex code. The paper introduces a static analysis approach at the LLVM IR level, proposing a feature-based LLVM Pass to identify core VM structures—Dispatch Start, Handler blocks, and VM Start/End—without executing the code. The method targets Tigress-obfuscated code and demonstrates detectability of core structures across switch, direct, and indirect virtualization modes in the absence of compiler optimizations. By encoding the detected features as dummy IR calls, the approach provides stable, executable-agnostic information to aid deobfuscation and could extend to other obfuscators, reducing reliance on dynamic traces.

Abstract

Malware often uses obfuscation to hinder security analysis. Among these techniques, virtualization-based obfuscation is particularly strong because it protects programs by translating original instructions into attacker-defined virtual machine (VM) bytecode, producing long and complex code that is difficult to analyze and deobfuscate. This paper aims to identify the structural components of virtualization-based obfuscation through static analysis. By examining the execution model of obfuscated code, we define and detect the key elements required for deobfuscation-namely the dispatch routine, handler blocks, and the VM region-using LLVM IR. Experimental results show that, in the absence of compiler optimizations, the proposed LLVM Pass successfully detects all core structures across major virtualization options, including switch, direct, and indirect modes.

Figures (6)

• Figure 1: Control Flow after VM-based obfuscation
• Figure 2: LLVM Framework structure llvm
• Figure 3: VM-obfuscated C code with switch option (in part)
• Figure 4: VM-obfuscated C code with direct option (in part)
• Figure 5: VM-obfuscated C code with indirect option (in part)