LibIHT: A Hardware-Based Approach to Efficient and Evasion-Resistant Dynamic Binary Analysis

TL;DR

LibIHT presents a hardware-assisted dynamic analysis framework that leverages Intel LBR and BTS to capture control-flow data with minimal overhead and improved evasion resistance. Implemented as a kernel module with a user-space CFG reconstruction library, it trades some semantic detail for speed and stealth, yet achieves near-perfect CFG fidelity on benign workloads. In adversarial settings, LibIHT remains largely undetected by anti-analysis checks while maintaining strong block coverage and keeping overhead well below software instrumentation tools. This work suggests hardware-centric tracing as a viable foundation for efficient, robust dynamic analysis in malware research and debugging pipelines.

Abstract

Dynamic program analysis is invaluable for malware detection, debugging, and performance profiling. However, software-based instrumentation incurs high overhead and can be evaded by anti-analysis techniques. In this paper, we propose LibIHT, a hardware-assisted tracing framework that leverages on-CPU branch tracing features (Intel Last Branch Record and Branch Trace Store) to efficiently capture program control-flow with minimal performance impact. Our approach reconstructs control-flow graphs (CFGs) by collecting hardware generated branch execution data in the kernel, preserving program behavior against evasive malware. We implement LibIHT as an OS kernel module and user-space library, and evaluate it on both benign benchmark programs and adversarial anti-instrumentation samples. Our results indicate that LibIHT reduces runtime overhead by over 150x compared to Intel Pin (7x vs 1,053x slowdowns), while achieving high fidelity in CFG reconstruction (capturing over 99% of execution basic blocks and edges). Although this hardware-assisted approach sacrifices the richer semantic detail available from full software instrumentation by capturing only branch addresses, this trade-off is acceptable for many applications where performance and low detectability are paramount. Our findings show that hardware-based tracing captures control flow information significantly faster, reduces detection risk and performs dynamic analysis with minimal interference.

Figures (7)

• Figure 1: High-level architecture of the Intel Pin (adapted from luk_pin_2005). The tool inserts a virtual machine, JIT compiler, and dispatcher to dynamically transform the target program, storing instrumented code in a dedicated code cache.
• Figure 2: Overview of DynamoRIO’s architecture (adapted from bruening_efficient_2004). Similar to Intel Pin, it uses a code cache for instrumented basic blocks, with a dispatcher managing control transfers to and from the running application.
• Figure 3: High-level architecture of Intel's LBR and BTS mechanism. LBR maintains a stack of MSRs on each CPU core for the most recent branches, while BTS extends this by logging branch records into a dedicated buffer in memory.
• Figure 4: High-level architecture of LibIHT. The kernel-space component manages hardware tracing and collects data for the target process and its children. An optional user-space monitor (e.g., a debugger or custom tool) can analyze the trace data, though it is not required for core operation.
• Figure 5: Comparison of mean execution times (in seconds) for standard Unix command‐line utilities under six execution modes.