The Reversing Machine: Reconstructing Memory Assumptions

TL;DR

The Reversing Machine (TRM) tackles the challenge of stealthy kernel- and user-space rootkits by delivering a hypervisor-based memory introspection framework that reconstructs memory layouts and fingerprints evasive malware in real time. It combines two novel techniques—suspended-process-based binary hooking for memory introspection and Mode-Based Execution Control (MBEC)—to transparently capture full memory traces and detect kernel/user-mode transitions, enabling end-to-end structure reconstruction and memory-analysis-driven signatures. The work provides a systematic memory-layout reconstruction pipeline, a memory-analyzer with a Longest Common Memory Address Pattern (LCMAP) approach for signature generation, and extensive evaluations showing speedups in reverse engineering (average ~{75}\% on kernel-structure reconstruction) and robust detection of obfuscated and packed malware that challenges traditional AV tools. Real-world demonstrations include a modified rootkit bypassing auditing tools and widespread AV-detection gaps, underscoring TRM’s potential to augment security research, malware analysis, and black-box software similarity assessment with hardware-facilitated memory traces. Overall, TRM offers a practical, hardware-backed memory introspection framework capable of reconstructing complex kernel structures, detecting sophisticated evasion techniques, and supporting cross-variant malware analysis with high fidelity $- including a measured speedup of $75\%$ in structure reconstruction and robust malware-signature capabilities across obfuscated and packed samples.

Abstract

Existing anti-malware software and reverse engineering toolkits struggle with stealthy sub-OS rootkits due to limitations of run-time kernel-level monitoring. A malicious kernel-level driver can bypass OS-level anti-virus mechanisms easily. Although static analysis of such malware is possible, obfuscation and packing techniques complicate offline analysis. Moreover, current dynamic analyzers suffer from virtualization performance overhead and create detectable traces that allow modern malware to evade them. To address these issues, we present \textit{The Reversing Machine} (TRM), a new hypervisor-based memory introspection design for reverse engineering, reconstructing memory offsets, and fingerprinting evasive and obfuscated user-level and kernel-level malware. TRM proposes two novel techniques that enable efficient and transparent analysis of evasive malware: hooking a binary using suspended process creation for hypervisor-based memory introspection, and leveraging Mode-Based Execution Control (MBEC) to detect user/kernel mode transitions and memory access patterns. Unlike existing malware detection environments, TRM can extract full memory traces in user and kernel spaces and hook the entire target memory map to reconstruct arrays, structures within the operating system, and possible rootkits. We perform TRM-assisted reverse engineering of kernel-level structures and show that it can speed up manual reverse engineering by 75\% on average. We obfuscate known malware with the latest packing tools and successfully perform similarity detection. Furthermore, we demonstrate a real-world attack by deploying a modified rootkit onto a driver that bypasses state-of-the-art security auditing tools. We show that TRM can detect each threat and that, out of 24 state-of-the-art AV solutions, only TRM can detect the most advanced threats.

Figures (7)

• Figure 1: High-level overview of TRM’s sub-systems and execution flow.
• Figure 2: The diagram of user-mode and kernel-mode transition detection in TRM.
• Figure 3: Intercepting entry points in TRM.
• Figure 4: The process of reconstructing structures.
• Figure 5: The code before and after modification.