Comparative Insights on Adversarial Machine Learning from Industry and Academia: A User-Study Approach

TL;DR

The paper addresses AML threats in ML by contrasting industry practitioner concerns with academic learning through two studies. It combines an online survey of 12 professionals with a Capture-the-Flag educational approach featuring two AML-focused challenges on NLP data poisoning and chatbot data-loop poisoning to gauge engagement and learning. Key findings reveal an expertise-related effect on AML concern, limited adoption of CTFs for AML education among professionals, and varied perceptions of ML-security integration, suggesting CTFs can educate when well designed and integrated with security curricula. The authors propose practical steps—adversarial training, data preprocessing, CIA/STRIDE integration, ethical governance, and a roadmap for CTF-based AML education—to bridge industry need and academic research in AML security. These insights advance AML defense education by offering dual-method evidence and a testbed for hands-on, cross-disciplinary security training in ML contexts, with clear implications for curricula and industry practice.$

Abstract

An exponential growth of Machine Learning and its Generative AI applications brings with it significant security challenges, often referred to as Adversarial Machine Learning (AML). In this paper, we conducted two comprehensive studies to explore the perspectives of industry professionals and students on different AML vulnerabilities and their educational strategies. In our first study, we conducted an online survey with professionals revealing a notable correlation between cybersecurity education and concern for AML threats. For our second study, we developed two CTF challenges that implement Natural Language Processing and Generative AI concepts and demonstrate a poisoning attack on the training data set. The effectiveness of these challenges was evaluated by surveying undergraduate and graduate students at Carnegie Mellon University, finding that a CTF-based approach effectively engages interest in AML threats. Based on the responses of the participants in our research, we provide detailed recommendations emphasizing the critical need for integrated security education within the ML curriculum.

Figures (9)

• Figure 1: User Interface for Challenge 1
• Figure 2: User Interface for Challenge 2
• Figure 3: Perception of participants before attempting CTF challenges
• Figure 4: Perception of participants after attempting CTF challenges
• Figure 5: Difficulty of the Challenges