ROPNN: Detection of ROP Payloads Using Deep Neural Networks
Xusheng Li, Zhisheng Hu, Haizhou Wang, Yiwei Fu, Ping Chen, Minghui Zhu, Peng Liu
TL;DR
This work tackles the challenge of detecting ROP payloads with high accuracy and low overhead by introducing ROPNN, an IDS that couples ASL-guided disassembly with a CNN classifier. It trains on gadget-chain-like instruction sequences derived from benign data and real gadget chains, enabling end-to-end detection without instrumenting protected programs. The approach achieves near-perfect detection (≈99.3%) with extremely low false positives (≈0.01%), detects all real-world exploits including those bypassing Bin-CFI, and imposes no runtime overhead on protected programs. Practically, ROPNN demonstrates strong generalization across multiple server applications and offers a scalable, deployable defense that complements traditional CFG-centric approaches like CFI.
Abstract
Return-oriented programming (ROP) is a code reuse attack that chains short snippets of existing code to perform arbitrary operations on target machines. Existing detection methods against ROP exhibit unsatisfactory detection accuracy and/or have high runtime overhead. In this paper, we present ROPNN, which innovatively combines address space layout guided disassembly and deep neural networks to detect ROP payloads. The disassembler treats application input data as code pointers and aims to find any potential gadget chains, which are then classified by a deep neural network as benign or malicious. Our experiments show that ROPNN has high detection rate (99.3%) and a very low false positive rate (0.01%). ROPNN successfully detects all of the 100 real-world ROP exploits that are collected in-the-wild, created manually or created by ROP exploit generation tools. Additionally, ROPNN detects all 10 ROP exploits that can bypass Bin-CFI. ROPNN is non-intrusive and does not incur any runtime overhead to the protected program.