GPT Store Mining and Analysis

TL;DR

The paper conducts a measurement study of the GPT Store ecosystem, addressing how GPTs are categorized, what drives their featured status, and the security risks they pose. It analyzes OpenAI's official store alongside three major third-party platforms, comparing category schemes, engagement signals, and update dynamics, then conducts a large-scale security assessment across 1,000 GPTs to quantify risks such as prompt attacks and data leakage. Key findings reveal mismatches and ambiguities in top-level categories, strong engagement signals linked to update frequency, and substantial vulnerabilities in prompts, files, and policy adherence. The work highlights practical implications for platform designers, developers, and policymakers, advocating improved taxonomy, richer GPT descriptions, user feedback systems, and strengthened defenses against manipulation and abuse in AI marketplaces.

Abstract

As a pivotal extension of the renowned ChatGPT, the GPT Store serves as a dynamic marketplace for various Generative Pre-trained Transformer (GPT) models, shaping the frontier of conversational AI. This paper presents an in-depth measurement study of the GPT Store, with a focus on the categorization of GPTs by topic, factors influencing GPT popularity, and the potential security risks. Our investigation starts with assessing the categorization of GPTs in the GPT Store, analyzing how they are organized by topics, and evaluating the effectiveness of the classification system. We then examine the factors that affect the popularity of specific GPTs, looking into user preferences, algorithmic influences, and market trends. Finally, the study delves into the security risks of the GPT Store, identifying potential threats and evaluating the robustness of existing security measures. This study offers a detailed overview of the GPT Store's current state, shedding light on its operational dynamics and user interaction patterns. Our findings aim to enhance understanding of the GPT ecosystem, providing valuable insights for future research, development, and policy-making in generative AI.

Figures (14)

• Figure 1: Distribution of GPT categories in OpenAI's GPT Store and GPTs Hunter.
• Figure 2: Categories of GPTs in GPTStore.AI.
• Figure 3: Categories of GPTs in GPTs App.
• Figure 4: Boxplot illustrating the Spearman correlation coefficient between dialogue volume and the number of ratings for GPTs in the GPT Store.
• Figure 5: Illustration of prompt jailbreaking process.