DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios

TL;DR

DecompileBench addresses the fragmentation in decompiler evaluation by providing a production-grade, three-pronged framework that combines real-world function extraction from OSS-Fuzz, runtime-aware validation, and automated LLM-based assessment of code quality. It reveals trade-offs between traditional decompilers and LLM-enhanced approaches: while LLMs improve readability and usefulness, they often compromise functional correctness, whereas traditional tools excel in semantic fidelity but lag in understandability. The framework offers three reports—compiler, runtime, and code quality—and uses Elo-based, LLM-guided judgments to quantify performance across 12 readability/helpfulness criteria, validated against human ratings with substantial agreement ($\kappa=0.778$). By open-sourcing DecompileBench, the authors enable researchers and security practitioners to compare tools under realistic conditions and guide tool selection based on task-specific needs.

Abstract

Decompilers are fundamental tools for critical security tasks, from vulnerability discovery to malware analysis, yet their evaluation remains fragmented. Existing approaches primarily focus on syntactic correctness through synthetic micro-benchmarks or subjective human ratings, failing to address real-world requirements for semantic fidelity and analyst usability. We present DecompileBench, the first comprehensive framework that enables effective evaluation of decompilers in reverse engineering workflows through three key components: \textit{real-world function extraction} (comprising 23,400 functions from 130 real-world programs), \textit{runtime-aware validation}, and \textit{automated human-centric assessment} using LLM-as-Judge to quantify the effectiveness of decompilers in reverse engineering workflows. Through a systematic comparison between six industrial-strength decompilers and six recent LLM-powered approaches, we demonstrate that LLM-based methods surpass commercial tools in code understandability despite 52.2% lower functionality correctness. These findings highlight the potential of LLM-based approaches to transform human-centric reverse engineering. We open source \href{https://github.com/Jennieett/DecompileBench}{DecompileBench} to provide a framework to advance research on decompilers and assist security experts in making informed tool selections based on their specific requirements.

Figures (9)

• Figure 1: Overview of dataset construction process.
• Figure 2: Three-dimensional evaluation framework for decompiler assessment: Successful recompilation rate, Runtime behavior consistency, and LLM-assessed code quality.
• Figure 3: Comparison of code quality across twelve dimensions using Elo scores. The average Elo score across all dimensions is shown in the bottom legend. The scores are relative within each dimension, with higher scores indicating a higher win rate. Note that absolute scores across different dimensions are not directly comparable.
• Figure 4: Output of Hex-Rays and GPT-4o. The error reports highlight where one decompiler fails and the other succeeds, demonstrating their pros and cons.
• Figure 5: Function prologue patching to redirect execution to a fixed memory-mapped location.