Better Privilege Separation for Agents by Restricting Data Types

TL;DR

Prompt injections threaten LLM-driven AI agents by allowing untrusted data to alter actions. The authors propose type-directed privilege separation, which restricts data flowing to the action-selection stage to a curated set of data types (e.g., integers, booleans, enums) while keeping broader input handling by a quarantined LLM. Across three case studies—online shopping, calendar scheduling, and bug fixing—the method eliminates prompt injection attacks (ASR = 0%) and preserves or minimally reduces utility, though some tasks incur a utility cost due to loss of freeform context. The approach is model-agnostic, compatible with any LLM, and aims to offer secure-by-design deployment for agents operating in untrusted environments, with plans to release source code for broader adoption.

Abstract

Large language models (LLMs) have become increasingly popular due to their ability to interact with unstructured content. As such, LLMs are now a key driver behind the automation of language processing systems, such as AI agents. Unfortunately, these advantages have come with a vulnerability to prompt injections, an attack where an adversary subverts the LLM's intended functionality with an injected task. Past approaches have proposed detectors and finetuning to provide robustness, but these techniques are vulnerable to adaptive attacks or cannot be used with state-of-the-art models. To this end we propose type-directed privilege separation for LLMs, a method that systematically prevents prompt injections. We restrict the ability of an LLM to interact with third-party data by converting untrusted content to a curated set of data types; unlike raw strings, each data type is limited in scope and content, eliminating the possibility for prompt injections. We evaluate our method across several case studies and find that designs leveraging our principles can systematically prevent prompt injection attacks while maintaining high utility.

Figures (5)

• Figure 1: Type-directed privilege separation for LLM agents. We illustrate the approach with a bug fixing agent. An undefended agent with unrestricted access to the list of open issues will be vulnerable to prompt injection (leftmost panel). The Dual LLM pattern improves security, but precludes the privileged LLM from accessing the issues list (middle panel). Our method allows the privileged agent to access context through a restricted set of data types (rightmost panel).
• Figure 2: An online shopping agent. The agent first generates a search query corresponding to the user instruction and then navigates the website to find the target product. User reviews present a vector for prompt injection.
• Figure 3: Our defended online shopping agent, which has been protected using type-directed privilege separation. Summarizing each review into a set of two integers prevents prompt injection.
• Figure 4: The calendar invitation agent: (a) An undefended agent, which is vulnerable to prompt injection in email replies and (b) Our defended agent, which converts the email chain to a set of candidate meeting times, preventing prompt injection.
• Figure 5: The coding agent: (a) The undefended agent, which is vulnerable to prompt injection in issue texts, and (b) The defended agent, in which the quarantined agent localizes the bug from the issue text and sends a safe handoff to the privileged agent to construct a fix.