Pop Quiz! Can a Large Language Model Help With Reverse Engineering?

TL;DR

This work evaluates whether a leading large language model can assist reverse engineering in zero-shot settings, including decompiled code, by prompting it to identify program purpose, capabilities, and key variables via a Q&A quiz framework. It introduces a structured experimental pipeline with true/false and open-ended information-extraction tasks across cybersecurity and ICS domains, plus randomized and decompiled code variants. Across 136,260 questions, the model produced 72,754 correct answers, signaling notable capability but not sufficiency for zero-shot deployment in real reverse-engineering workflows. The study systematically analyzes prompt design, parameter tuning, and the impact of code obfuscation, and discusses limitations and potential enhancements, such as embedding-based representations or domain-specific augmentation, to bolster practical utility.

Abstract

Large language models (such as OpenAI's Codex) have demonstrated impressive zero-shot multi-task capabilities in the software domain, including code explanation. In this work, we examine if this ability can be used to help with reverse engineering. Specifically, we investigate prompting Codex to identify the purpose, capabilities, and important variable names or values from code, even when the code is produced through decompilation. Alongside an examination of the model's responses in answering open-ended questions, we devise a true/false quiz framework to characterize the performance of the language model. We present an extensive quantitative analysis of the measured performance of the language model on a set of program purpose identification and information extraction tasks: of the 136,260 questions we posed, it answered 72,754 correctly. A key takeaway is that while promising, LLMs are not yet ready for zero-shot reverse engineering.

Figures (19)

• Figure 1: Reverse engineering inquiry evaluation system.
• Figure 2: C 'malware' delete_listen which deletes files from directory paths received via open socket, presented in a 'Q&A' template for use with the code-davinci-001LLM.
• Figure 3: Randomized version of delete_listen from \ref{['fig:lst:prog-delete-listen']}.
• Figure 4: Decompiled (using gcc -o1) delete_listen from \ref{['fig:lst:prog-delete-listen']} (with some reformatting).
• Figure 5: ICS pid_d program listing (some reformatting).