Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Benchmarking Design Pattern Detection Under Obfuscation: Reproducing and Evaluating Attention-Based Detection Method

Manthan Shenoy, Andreas Rausch

TL;DR

This work interrogates whether attention-based design pattern detectors truly understand software semantics or rely on superficial syntactic cues. By reproducing DPDAtt and curating a semantically preserved yet identifier-obfuscated 34-file Java subset across 13 GoF patterns, the authors demonstrate that removing naming signals causes a dramatic drop in detection performance. The results reveal a strong dependence on token-level cues and offer a minimal, reproducible benchmark to assess semantic robustness of pattern detectors. The proposed Obfuscated Corpus serves as a practical probe for evaluating true semantic generalization in design pattern detection tools, guiding future development toward semantically aware methods.

Abstract

This paper investigates the semantic robustness of attention-based classifiers for design pattern detection, particularly focusing on their reliance on structural and behavioral semantics. We reproduce the DPDAtt, an attention-based design pattern detection approach using learning-based classifiers, and evaluate its performance under obfuscation. To this end, we curate an obfuscated version of the DPDAtt Corpus, where the name identifiers in code such as class names, method names, etc., and string literals like print statements and comment blocks are replaced while preserving control flow, inheritance, and logic. Our findings reveal that these trained classifiers in DPDAtt depend significantly on superficial syntactic features, leading to substantial misclassification when such cues are removed through obfuscation. This work highlights the need for more robust detection tools capable of capturing deeper semantic meanings in source code. We propose our curated Obfuscated corpus (containing 34 Java source files) as a reusable proof-of-concept benchmark for evaluating state-of-the-art design pattern detectors on their true semantic generalization capabilities.

Towards Benchmarking Design Pattern Detection Under Obfuscation: Reproducing and Evaluating Attention-Based Detection Method

TL;DR

This work interrogates whether attention-based design pattern detectors truly understand software semantics or rely on superficial syntactic cues. By reproducing DPDAtt and curating a semantically preserved yet identifier-obfuscated 34-file Java subset across 13 GoF patterns, the authors demonstrate that removing naming signals causes a dramatic drop in detection performance. The results reveal a strong dependence on token-level cues and offer a minimal, reproducible benchmark to assess semantic robustness of pattern detectors. The proposed Obfuscated Corpus serves as a practical probe for evaluating true semantic generalization in design pattern detection tools, guiding future development toward semantically aware methods.

Abstract

This paper investigates the semantic robustness of attention-based classifiers for design pattern detection, particularly focusing on their reliance on structural and behavioral semantics. We reproduce the DPDAtt, an attention-based design pattern detection approach using learning-based classifiers, and evaluate its performance under obfuscation. To this end, we curate an obfuscated version of the DPDAtt Corpus, where the name identifiers in code such as class names, method names, etc., and string literals like print statements and comment blocks are replaced while preserving control flow, inheritance, and logic. Our findings reveal that these trained classifiers in DPDAtt depend significantly on superficial syntactic features, leading to substantial misclassification when such cues are removed through obfuscation. This work highlights the need for more robust detection tools capable of capturing deeper semantic meanings in source code. We propose our curated Obfuscated corpus (containing 34 Java source files) as a reusable proof-of-concept benchmark for evaluating state-of-the-art design pattern detectors on their true semantic generalization capabilities.

Paper Structure

This paper contains 23 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Intro to the DPDAtt mzid2024attentionApproach
  4. Description of Our experimental workflow
  5. Description of Our experimental workflow
  6. Obfuscation Tool Description
  7. Obfuscation Strategy and Objectives
  8. Obfuscation Tool selection and comparison
  9. Obfuscated Corpus preparation, Manual Stubbing, and Compilation
  10. Obfuscation and De-compilation
  11. Final Test Set: Obfuscated corpus
  12. Training the Classifiers of DPDAtt-Baseline for Pattern Detection
  13. Dataset and Pre-processing
  14. Classifier Setup
  15. Training Procedure
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Overview of our experimental workflow
  • Figure 2: One of the 34 original file sample EmSingleton.java loaded and compiled in Eclipse IDE (prior to obfuscation).