"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice

TL;DR

Bridging the gap between adversarial ML research and practice, the paper frames ML security as a system-wide problem with threat models comprising $goal$, $knowledge$, $capabilities$, and $strategy$. It uses three real-world case studies and a survey of 88 recent top-venue papers to reveal practical gaps, such as overreliance on neural models, limited real-system evaluation, and sparse economics consideration. Four actionable positions are proposed: adapt threat models to entire ML systems, adopt cost-driven security assessments, foster industry–academia collaboration, and enforce just culture with reproducible research. The authors argue that adopting these measures will increase the real-world impact of adversarial ML work by aligning research with production security needs and economics.

Abstract

Recent years have seen a proliferation of research on adversarial machine learning. Numerous papers demonstrate powerful algorithmic attacks against a wide variety of machine learning (ML) models, and numerous other papers propose defenses that can withstand most attacks. However, abundant real-world evidence suggests that actual attackers use simple tactics to subvert ML-driven systems, and as a result security practitioners have not prioritized adversarial ML defenses. Motivated by the apparent gap between researchers and practitioners, this position paper aims to bridge the two domains. We first present three real-world case studies from which we can glean practical insights unknown or neglected in research. Next we analyze all adversarial ML papers recently published in top security conferences, highlighting positive trends and blind spots. Finally, we state positions on precise and cost-driven threat modeling, collaboration between industry and academia, and reproducible research. We believe that our positions, if adopted, will increase the real-world impact of future endeavours in adversarial ML, bringing both researchers and practitioners closer to their shared goal of improving the security of ML systems.

Figures (13)

• Figure 1: An ML system. The system receives an input, which is preprocessed and then fed to an ML model, the results of which may be further processed before providing the system's final output.
• Figure 2: An invisible ML system. The low privileged users in the environment interact with the ML system, the output of which can be used in diverse ways by its managers. For example, the alarms of a NIDS may be inspected by sysadmins and then used to block suspicious hosts; whereas a network management system can dynamically allocate available resources (e.g., bandwidth). In some cases the output does not trigger any immediate action.
• Figure 3: Example of Facebook's ML system for spam detection. The system consists of a "funnel" of four interconnected defensive layers, each with its own logic. The attacker must bypass all layers to be successful.
• Figure 4: Four evasive phishing samples, depicting the use of masking, cropping, blurring, and misspellings to disrupt the detection of company names, logos, and login-related keywords (e.g., "Passwrd").
• Figure 5: Temporal distribution of cumulative submissions (y-axis) during the phishing MLSEC in 2021 (started on Aug. 6th). Each line indicates a team (q=queries). A detailed explanation of this figure is in Appendix \ref{['sapp:clarification']}.