Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs

David Rodriguez, William Seymour, Jose M. Del Alamo, Jose Such

TL;DR

The paper tackles the challenge of enforcing policy compliance for user-configured GPTs in a rapidly growing GPT Store. It introduces a fully automated, black-box framework that discovers Custom GPTs, generates policy-aligned red-teaming prompts, and uses an LLM as a judge to assess compliance, validated against human annotations (F1 ≈ 0.975) and applied to 782 GPTs. The large-scale study reveals that 58.7% of evaluated GPTs violate at least one policy, with domain variation: romantic GPTs are highly non-compliant while cybersecurity-focused GPTs are largely compliant; most violations resemble base-model behaviors and are amplified by customization. The work demonstrates the feasibility and cost-effectiveness of behavior-based policy evaluation at scale, exposes governance gaps in current GPT Store reviews, and discusses implications for platform safety, responsible disclosure, and broader applicability to other LLM ecosystems.

Abstract

User-configured chatbots built on top of large language models are increasingly available through centralized marketplaces such as OpenAI's GPT Store. While these platforms enforce usage policies intended to prevent harmful or inappropriate behavior, the scale and opacity of customized chatbots make systematic policy enforcement challenging. As a result, policy-violating chatbots continue to remain publicly accessible despite existing review processes. This paper presents a fully automated method for evaluating the compliance of Custom GPTs with its marketplace usage policy using black-box interaction. The method combines large-scale GPT discovery, policy-driven red-teaming prompts, and automated compliance assessment using an LLM-as-a-judge. We focus on three policy-relevant domains explicitly addressed in OpenAI's usage policies: Romantic, Cybersecurity, and Academic GPTs. We validate our compliance assessment component against a human-annotated ground-truth dataset, achieving an F1 score of 0.975 for binary policy violation detection. We then apply the method in a large-scale empirical study of 782 Custom GPTs retrieved from the GPT Store. The results show that 58.7% of the evaluated GPTs exhibit at least one policy-violating response, with substantial variation across policy domains. A comparison with the base models (GPT-4 and GPT-4o) indicates that most violations originate from model-level behavior, while customization tends to amplify these tendencies rather than create new failure modes. Our findings reveal limitations in current review mechanisms for user-configured chatbots and demonstrate the feasibility of scalable, behavior-based policy compliance evaluation.

Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs

TL;DR

The paper tackles the challenge of enforcing policy compliance for user-configured GPTs in a rapidly growing GPT Store. It introduces a fully automated, black-box framework that discovers Custom GPTs, generates policy-aligned red-teaming prompts, and uses an LLM as a judge to assess compliance, validated against human annotations (F1 ≈ 0.975) and applied to 782 GPTs. The large-scale study reveals that 58.7% of evaluated GPTs violate at least one policy, with domain variation: romantic GPTs are highly non-compliant while cybersecurity-focused GPTs are largely compliant; most violations resemble base-model behaviors and are amplified by customization. The work demonstrates the feasibility and cost-effectiveness of behavior-based policy evaluation at scale, exposes governance gaps in current GPT Store reviews, and discusses implications for platform safety, responsible disclosure, and broader applicability to other LLM ecosystems.

Abstract

User-configured chatbots built on top of large language models are increasingly available through centralized marketplaces such as OpenAI's GPT Store. While these platforms enforce usage policies intended to prevent harmful or inappropriate behavior, the scale and opacity of customized chatbots make systematic policy enforcement challenging. As a result, policy-violating chatbots continue to remain publicly accessible despite existing review processes. This paper presents a fully automated method for evaluating the compliance of Custom GPTs with its marketplace usage policy using black-box interaction. The method combines large-scale GPT discovery, policy-driven red-teaming prompts, and automated compliance assessment using an LLM-as-a-judge. We focus on three policy-relevant domains explicitly addressed in OpenAI's usage policies: Romantic, Cybersecurity, and Academic GPTs. We validate our compliance assessment component against a human-annotated ground-truth dataset, achieving an F1 score of 0.975 for binary policy violation detection. We then apply the method in a large-scale empirical study of 782 Custom GPTs retrieved from the GPT Store. The results show that 58.7% of the evaluated GPTs exhibit at least one policy-violating response, with substantial variation across policy domains. A comparison with the base models (GPT-4 and GPT-4o) indicates that most violations originate from model-level behavior, while customization tends to amplify these tendencies rather than create new failure modes. Our findings reveal limitations in current review mechanisms for user-configured chatbots and demonstrate the feasibility of scalable, behavior-based policy compliance evaluation.

Paper Structure

This paper contains 49 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Problem Context
  3. Custom GPTs and the GPT Store
  4. Policy Compliance
  5. Defining Policy Compliance
  6. Compliance vs. Safety and Alignment
  7. Challenges in Evaluating Custom GPT Compliance
  8. Safety Evaluation and Red-Teaming
  9. Related Work
  10. Policy Compliance Evaluation Method
  11. Overview
  12. Components of the Evaluation Method
  13. GPT Collector & Interactor
  14. Red-Teaming Prompts Generator
  15. Compliance Assessment
  16. ...and 34 more sections

Figures (4)

  • Figure 1: Search of “girlfriend” keyword in the GPT store, showcasing the proliferation of models that violate OpenAI's usage policies explicitly prohibiting romantic companionship.
  • Figure 2: Overview of the policy compliance evaluation method for Custom GPTs. The process spans five stages, from GPT retrieval to storage of compliance results, and is coordinated by the Orchestrator.
  • Figure 3: Characteristics of Evaluated Custom GPTs. Note that the total count of chats and ratings does not align with the number of Custom GPTs in the dataset, as some GPTs lack this data in the GPT store, likely due to having zero recorded interactions or being recently published.
  • Figure 4: Compliance and Non-Compliance Ratios Across GPT Categories. This figure illustrates the compliance rates for three major Custom GPT categories: Academic, Cybersecurity, and Romantic. Evaluation was conducted based on adherence to OpenAI’s usage policies. The results highlight a significant variation among categories, with Romantic GPTs exhibiting the highest non-compliance rate (98.0%) and Cybersecurity GPTs demonstrating the highest compliance rate (92.6%).