BinPRE: Enhancing Field Inference in Binary Analysis Based Protocol Reverse Engineering

TL;DR

BinPRE presents a binary-analysis–based PRE tool that significantly improves field inference by (1) using instruction-based semantic similarity for robust format extraction, (2) deploying a library of atomic semantic detectors for richer semantic inference, and (3) applying a cluster-and-refine paradigm to leverage contextual information. The approach yields higher format perfection and semantic F1-scores than state-of-the-art baselines, and demonstrably enhances downstream protocol fuzzing performance, including discovery of a zero-day vulnerability. It also provides reproducible baselines by re-implementing prior tools and releasing its codebase publicly. The work advances practical PRE for diverse protocols and opens pathways for more reliable protocol auditing and security testing.

Abstract

Protocol reverse engineering (PRE) aims to infer the specification of network protocols when the source code is not available. Specifically, field inference is one crucial step in PRE to infer the field formats and semantics. To perform field inference, binary analysis based PRE techniques are one major approach category. However, such techniques face two key challenges - (1) the format inference is fragile when the logics of processing input messages may vary among different protocol implementations, and (2) the semantic inference is limited by inadequate and inaccurate inference rules. To tackle these challenges, we present BinPRE, a binary analysis based PRE tool. BinPRE incorporates (1) an instruction-based semantic similarity analysis strategy for format extraction; (2) a novel library composed of atomic semantic detectors for improving semantic inference adequacy; and (3) a cluster-and-refine paradigm to further improve semantic inference accuracy. We have evaluated BinPRE against five existing PRE tools, including Polyglot, AutoFormat, Tupni, BinaryInferno and DynPRE. The evaluation results on eight widely-used protocols show that BinPRE outperforms the prior PRE tools in both format and semantic inference. BinPRE achieves the perfection of 0.73 on format extraction and the F1-score of 0.74 (0.81) on semantic inference of types (functions), respectively. The field inference results of BinPRE have helped improve the effectiveness of protocol fuzzing by achieving 5-29% higher branch coverage, compared to those of the best prior PRE tool. BinPRE has also helped discover one new zero-day vulnerability, which otherwise cannot be found.

Figures (13)

• Figure 1: An example message of DNP3.0 protocol (the fields with different semantics are annotated by different colors).
• Figure 2: BinPRE overview
• Figure 3: The process of analyzing a field that is prone to the over-segmentation error.
• Figure 4: The process of analyzing a field that is prone to the under-segmentation error.
• Figure 5: Assembly instructions for lines 16-28 of Listing \ref{['lst:listing1']}