Adversarial Vulnerability Transcends Computational Paradigms: Feature Engineering Provides No Defense Against Neural Adversarial Transfer

TL;DR

It is demonstrated that adversarial vulnerability is not an artifact of end-to-end differentiability but a fundamental property of image classification systems, with implications for security-critical deployments across computational paradigms.

Abstract

Deep neural networks are vulnerable to adversarial examples--inputs with imperceptible perturbations causing misclassification. While adversarial transfer within neural networks is well-documented, whether classical ML pipelines using handcrafted features inherit this vulnerability when attacked via neural surrogates remains unexplored. Feature engineering creates information bottlenecks through gradient quantization and spatial binning, potentially filtering high-frequency adversarial signals. We evaluate this hypothesis through the first comprehensive study of adversarial transfer from DNNs to HOG-based classifiers. Using VGG16 as a surrogate, we generate FGSM and PGD adversarial examples and test transfer to four classical classifiers (KNN, Decision Tree, Linear SVM, Kernel SVM) and a shallow neural network across eight HOG configurations on CIFAR-10. Our results strongly refute the protective hypothesis: all classifiers suffer 16.6%-59.1% relative accuracy drops, comparable to neural-to-neural transfer. More surprisingly, we discover attack hierarchy reversal--contrary to patterns where iterative PGD dominates FGSM within neural networks, FGSM causes greater degradation than PGD in 100% of classical ML cases, suggesting iterative attacks overfit to surrogate-specific features that don't survive feature extraction. Block normalization provides partial but insufficient mitigation. These findings demonstrate that adversarial vulnerability is not an artifact of end-to-end differentiability but a fundamental property of image classification systems, with implications for security-critical deployments across computational paradigms.

Figures (9)

• Figure 1: Cosine similarity distributions between original and adversarial images at $\epsilon = 8/255$ (worst-case perturbation budget). Mean similarities of 0.831 (FGSM) and 0.832 (PGD) confirm that adversarial perturbations preserve global image structure even at aggressive attack strengths.
• Figure 2: Sensitivity of HOG-based classifiers to adversarial perturbation magnitude across configurations. Shaded regions highlight the FGSM-PGD gap.
• Figure 3: Comparative analysis of adversarial robustness across feature-engineered and end-to-end classification pipelines. Classification accuracy of HOG-based pipelines (RBF-SVM, K-NN, Linear SVM, and ANN) compared against CNN transfer baselines (AlexNet as target, VGG as surrogate) under clean, FGSM, and PGD conditions. All HOG classifiers use Configuration C5 (block size = 3).
• Figure 4: FGSM vs. PGD transfer effectiveness to HOG feature space: evidence of attack hierarchy reversal. All points lie above the diagonal, indicating FGSM causes greater accuracy degradation than PGD across all configurations.
• Figure 5: Sensitivity of HOG-based classifiers to adversarial perturbation magnitude: $\epsilon = 4/255$ vs. $\epsilon = 8/255$.