LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Plugins

TL;DR

This paper tackles the security, privacy, and safety challenges of LLM platforms that support third-party plugins by creating a systematic attack taxonomy. It develops an actionable, extensible framework and applies it to OpenAI's plugin ecosystem, revealing numerous potential attacks across the three-party model (plugins, users, and the LLM platform). Through crawling and interacting with 268 OpenAI plugins, the authors identify concrete risks such as credential exfiltration, data harvesting, prompt hijacking, and server DoS, and iteratively refine the taxonomy accordingly. The work concludes with practical recommendations—like sandboxing, permission models, and ongoing threat modeling—to guide secure design of current and future LLM-based ecosystems. Overall, the framework provides a reality-grounded lens for researchers and practitioners to evaluate and strengthen the security posture of evolving LLM plugin platforms.

Abstract

Large language model (LLM) platforms, such as ChatGPT, have recently begun offering an app ecosystem to interface with third-party services on the internet. While these apps extend the capabilities of LLM platforms, they are developed by arbitrary third parties and thus cannot be implicitly trusted. Apps also interface with LLM platforms and users using natural language, which can have imprecise interpretations. In this paper, we propose a framework that lays a foundation for LLM platform designers to analyze and improve the security, privacy, and safety of current and future third-party integrated LLM platforms. Our framework is a formulation of an attack taxonomy that is developed by iteratively exploring how LLM platform stakeholders could leverage their capabilities and responsibilities to mount attacks against each other. As part of our iterative process, we apply our framework in the context of OpenAI's plugin (apps) ecosystem. We uncover plugins that concretely demonstrate the potential for the types of issues that we outline in our attack taxonomy. We conclude by discussing novel challenges and by providing recommendations to improve the security, privacy, and safety of present and future LLM-based computing platforms.

Figures (7)

• Figure 1: Life cycle of a user command to LLM that requires use of a plugin: User installs a plugin on LLM platform from the plugin store (step 1). Plugin description and its endpoints are fed to the LLM to build the context, necessary for interpreting user prompt (step 2). User makes a prompt to the LLM that requires the use of the installed plugin (step 3). LLM selects the relevant plugin based on its description (step 4) and makes a request to the plugin API endpoint with the required parameters (step 5). LLM then interprets the response from the plugin API endpoint and displays it to the user.
• Figure 2: User interaction with AutoInfra1 plugin.
• Figure 3: Dual presence of Upskillr plugin on the OpenAI plugin store.
• Figure 4: User interaction with PDF Exporter plugin.
• Figure 5: User interaction with ChatGPT, when AMZPRO is enabled but not used.