Countering Autonomous Cyber Threats

TL;DR

Using target machines from a commercial provider, the most recently released downloadable models are found to be on par with a leading proprietary model at conducting simple cyber attacks with common hacking tools against known vulnerabilities and the implications for AI safety and governance with respect to cybersecurity is analyzed.

Abstract

With the capability to write convincing and fluent natural language and generate code, Foundation Models present dual-use concerns broadly and within the cyber domain specifically. Generative AI has already begun to impact cyberspace through a broad illicit marketplace for assisting malware development and social engineering attacks through hundreds of malicious-AI-as-a-services tools. More alarming is that recent research has shown the potential for these advanced models to inform or independently execute offensive cyberspace operations. However, these previous investigations primarily focused on the threats posed by proprietary models due to the until recent lack of strong open-weight model and additionally leave the impacts of network defenses or potential countermeasures unexplored. Critically, understanding the aptitude of downloadable models to function as offensive cyber agents is vital given that they are far more difficult to govern and prevent their misuse. As such, this work evaluates several state-of-the-art FMs on their ability to compromise machines in an isolated network and investigates defensive mechanisms to defeat such AI-powered attacks. Using target machines from a commercial provider, the most recently released downloadable models are found to be on par with a leading proprietary model at conducting simple cyber attacks with common hacking tools against known vulnerabilities. To mitigate such LLM-powered threats, defensive prompt injection (DPI) payloads for disrupting the malicious cyber agent's workflow are demonstrated to be effective. From these results, the implications for AI safety and governance with respect to cybersecurity is analyzed.

Figures (5)

• Figure 1: An overview of the operating infrastructure used in this thesis. The offensive cyber agent runtime communicates to AI APIs such as OpenAI or Ollama and then parses actions to be executed within the Kali Linux environment. The Kali environment is hosted in an isolated setting within Docker, with targets either also hosted locally or remotely on HackTheBox, which is accessed via secure VPN.
• Figure 2: This figure depicts the communication flow between the AI inference server, the cyber agent logic, and the Kali Linux and Target machines. Since the agent runtime and the Kali instance are separate and connected by SSH, the Kali machine could be hosted either locally or remotely.
• Figure 3: The communication sequence diagram of a multi-stage defensive prompt injection. The SSH banner of the honeypot is modified to suggest that the machine is vulnerable to a non-existent exploit that can be retrieved from another defender-controlled server. The agent decides to retrieve the fake exploit script and execute it, resulting in the opening of a bind shell which would allow authorities to counter exploit and shut down the attacker's machine.
• Figure 4: An example of UK AISI's Inspect Framework, where it is being used to view trials of GPT-4o as it encounters Defensive Prompt Injections.
• Figure 5: The composition of the cyber agent used in the experiments, with brief descriptions of the tools and their interfaces.