ACE: A Security Architecture for LLM-Integrated App Systems

TL;DR

ACE addresses security gaps in LLM-integrated app systems by decoupling planning from execution through a three-phase architecture: abstract planning using trusted information, concrete planning with app matching and static policy verification, and isolated execution enforcing predefined security constraints. The approach uses a lattice-based information-flow model to statically verify plan safety, and an orchestrator-worker execution environment to enforce least privilege and prevent cross-app interference. Empirical evaluations demonstrate ACE's strong defense against indirect prompt injection benchmarks (InjecAgent, ASB) and high utility on realistic tool-use tasks, indicating practical viability for hardened LLM-driven agents. Overall, ACE offers a principled, security-by-design path for robust, auditable LLM-enabled systems with strict data-flow controls.

Abstract

LLM-integrated app systems extend the utility of Large Language Models (LLMs) with third-party apps that are invoked by a system LLM using interleaved planning and execution phases to answer user queries. These systems introduce new attack vectors where malicious apps can cause integrity violation of planning or execution, availability breakdown, or privacy compromise during execution. In this work, we identify new attacks impacting the integrity of planning, as well as the integrity and availability of execution in LLM-integrated apps, and demonstrate them against IsolateGPT, a recent solution designed to mitigate attacks from malicious apps. We propose Abstract-Concrete-Execute (ACE), a new secure architecture for LLM-integrated app systems that provides security guarantees for system planning and execution. Specifically, ACE decouples planning into two phases by first creating an abstract execution plan using only trusted information, and then mapping the abstract plan to a concrete plan using installed system apps. We verify that the plans generated by our system satisfy user-specified secure information flow constraints via static analysis on the structured plan output. During execution, ACE enforces data and capability barriers between apps, and ensures that the execution is conducted according to the trusted abstract plan. We show experimentally that ACE is secure against attacks from the InjecAgent and Agent Security Bench benchmarks for indirect prompt injection, and our newly introduced attacks. We also evaluate the utility of ACE in realistic environments, using the Tool Usage suite from the LangChain benchmark. Our architecture represents a significant advancement towards hardening LLM-based systems using system security principles.

Figures (13)

• Figure 1: Comparison of system architectures. In typical systems (left) a central system LLM is responsible for planning control flow based on the user queries and available system utilities. Planning and execution phases are interleaved, producing a control flow mechanism that is arbitrarily dependent on the user instructions, app descriptions, and intermediate system outputs. Our system ACE (right) generates a structured plan prior to execution based on trusted information.
• Figure 2: Illustration of Execution Flow Disruption attack on IsolateGPT, which causes an availability breakdown in execution.
• Figure 3: Overview of our three-phase ACE secure LLM-integrated app system architecture. First, our system generates an abstract plan using a set of abstract apps, generated using only fully-trusted query information. Next, we match abstract apps with concrete apps installed on the system in the concrete planning phase. Matching consists of a binary decision made independently between each pair of abstract and concrete app. Finally, the concrete plan is executed in a carefully managed execution environment which enforces isolation between system app instances.
• Figure 4: Example abstract plan for the user query "Load document 'file.txt' from my documents and summarize the contents." DocumentLoader and TextSummarizer are abstract apps automatically generated by the planner and are not affected by the apps installed on the system.
• Figure 5: An example abstract plan with information leakage present. Privileged information is loaded into the variable data from the app load_bank_details and subsequently passed to the uncleared location send_email. Static analysis detects the dependency and blocks the execution. It is assumed that the concrete plan matches send_email to a concrete app with clearance "personal" and load_bank_details to an app with clearance "financial".