Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic State Machine Inference for Binary Protocol Reverse Engineering

Junhai Yang, Fenghua Li, Yixuan Zhang, Junhao Zhang, Liang Fang, Yunchuan Guo

TL;DR

The paper tackles the challenge of protocol reverse engineering in mixed unknown traffic by targeting Protocol State Machine (PSM) inference. It introduces an automatic pipeline that first clusters protocol formats with a fuzzy membership-based auto-converging DBSCAN (ACDA), then clusters sessions via Needleman-Wunsch alignment and K-medoids, and finally refines probabilistic PSMs with total-probability-based noise suppression. Key contributions include the ACDA-based protocol format clustering, NW-KMedoids session clustering for mixed protocols, and a robust probabilistic PSM inference enhanced from Veritas. Experiments on TLSv1.2 and SMTP traces show high state/transition matching (SMC≈1.0, TMC≈0.86–1.0) and strong clustering performance, demonstrating effective PSM inference for both binary and text protocols in mixed environments.

Abstract

Protocol Reverse Engineering (PRE) is used to analyze protocols by inferring their structure and behavior. However, current PRE methods mainly focus on field identification within a single protocol and neglect Protocol State Machine (PSM) analysis in mixed protocol environments. This results in insufficient analysis of protocols' abnormal behavior and potential vulnerabilities, which are crucial for detecting and defending against new attack patterns. To address these challenges, we propose an automatic PSM inference framework for unknown protocols, including a fuzzy membership-based auto-converging DBSCAN algorithm for protocol format clustering, followed by a session clustering algorithm based on Needleman-Wunsch and K-Medoids algorithms to classify sessions by protocol type. Finally, we refine a probabilistic PSM algorithm to infer protocol states and the transition conditions between these states. Experimental results show that, compared with existing PRE techniques, our method can infer PSMs while enabling more precise classification of protocols.

Automatic State Machine Inference for Binary Protocol Reverse Engineering

TL;DR

The paper tackles the challenge of protocol reverse engineering in mixed unknown traffic by targeting Protocol State Machine (PSM) inference. It introduces an automatic pipeline that first clusters protocol formats with a fuzzy membership-based auto-converging DBSCAN (ACDA), then clusters sessions via Needleman-Wunsch alignment and K-medoids, and finally refines probabilistic PSMs with total-probability-based noise suppression. Key contributions include the ACDA-based protocol format clustering, NW-KMedoids session clustering for mixed protocols, and a robust probabilistic PSM inference enhanced from Veritas. Experiments on TLSv1.2 and SMTP traces show high state/transition matching (SMC≈1.0, TMC≈0.86–1.0) and strong clustering performance, demonstrating effective PSM inference for both binary and text protocols in mixed environments.

Abstract

Protocol Reverse Engineering (PRE) is used to analyze protocols by inferring their structure and behavior. However, current PRE methods mainly focus on field identification within a single protocol and neglect Protocol State Machine (PSM) analysis in mixed protocol environments. This results in insufficient analysis of protocols' abnormal behavior and potential vulnerabilities, which are crucial for detecting and defending against new attack patterns. To address these challenges, we propose an automatic PSM inference framework for unknown protocols, including a fuzzy membership-based auto-converging DBSCAN algorithm for protocol format clustering, followed by a session clustering algorithm based on Needleman-Wunsch and K-Medoids algorithms to classify sessions by protocol type. Finally, we refine a probabilistic PSM algorithm to infer protocol states and the transition conditions between these states. Experimental results show that, compared with existing PRE techniques, our method can infer PSMs while enabling more precise classification of protocols.

Paper Structure

This paper contains 18 sections, 10 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Data Preprocessing
  5. Protocol Format Clustering
  6. Session Clustering
  7. Protocol State Machine Inference
  8. Experiments and Analysis
  9. Environment and Dataset
  10. Metrics
  11. Clustering Evaluation
  12. PSM Inference Evaluation
  13. Functionality Comparison
  14. Experiment Results
  15. Protocol Format Clustering
  16. ...and 3 more sections

Figures (5)

  • Figure 1: System architecture of automatic PSM inference framework for unknown protocols
  • Figure 2: NW algorithm $PFC$ sequence alignment and similarity calculation.
  • Figure 3: The optimal clustering result calculation for the ACDA.
  • Figure 4: The optimal parameters selection
  • Figure 5: The process of PSM inference.