A LLM Assisted Exploitation of AI-Guardian

TL;DR

This work investigates whether GPT-4 can function as an autonomous attacker to break a contemporary adversarial defense, AI-Guardian. By implementing a three-stage extraction attack (mask, permutation, pattern) under a defined threat model, the authors demonstrate that AI-Guardian’s claimed robustness collapses in practice, reducing robustness from near 100% to around 8% on MNIST-like tasks and achieving over 90% targeted attack success on official defenses. The study highlights both the vulnerability of AI-Guardian and the potential for LLMs to accelerate real-world adversarial research, prompting a discussion on evaluation rigor and the evolving role of AI-assisted inquiry in security research. Overall, the results raise important questions about the reliability of defenses that rely on secret transformations and about how to properly validate robustness in future work.

Abstract

Large language models (LLMs) are now highly capable at a diverse range of tasks. This paper studies whether or not GPT-4, one such LLM, is capable of assisting researchers in the field of adversarial machine learning. As a case study, we evaluate the robustness of AI-Guardian, a recent defense to adversarial examples published at IEEE S&P 2023, a top computer security conference. We completely break this defense: the proposed scheme does not increase robustness compared to an undefended baseline. We write none of the code to attack this model, and instead prompt GPT-4 to implement all attack algorithms following our instructions and guidance. This process was surprisingly effective and efficient, with the language model at times producing code from ambiguous instructions faster than the author of this paper could have done. We conclude by discussing (1) the warning signs present in the evaluation that suggested to us AI-Guardian would be broken, and (2) our experience with designing attacks and performing novel research using the most recent advances in language modeling.