Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection

Gabriel Chua, Shing Yee Chan, Shaun Khoo

TL;DR

The paper tackles off-topic misuse in LLMs and the limitations of real-world data for guardrails. It introduces a flexible, data-free guardrail development methodology that uses qualitative problem analysis and LLM-driven synthetic data to train both bi-encoder and cross-encoder detectors, producing calibrated probability scores for threshold-based refusals. The approach demonstrates high discrimination (ROC-AUC near 0.99) and strong generalization to jailbreak and harmful prompts, with favorable inference speeds, and is openly released as synthetic data and models to accelerate safety research. Practical deployment considerations, calibration, and limitations are discussed to guide real-world adoption and future work in pre-production safety safeguards.

Abstract

Large Language Models (LLMs) are prone to off-topic misuse, where users may prompt these models to perform tasks beyond their intended scope. Current guardrails, which often rely on curated examples or custom classifiers, suffer from high false-positive rates, limited adaptability, and the impracticality of requiring real-world data that is not available in pre-production. In this paper, we introduce a flexible, data-free guardrail development methodology that addresses these challenges. By thoroughly defining the problem space qualitatively and passing this to an LLM to generate diverse prompts, we construct a synthetic dataset to benchmark and train off-topic guardrails that outperform heuristic approaches. Additionally, by framing the task as classifying whether the user prompt is relevant with respect to the system prompt, our guardrails effectively generalize to other misuse categories, including jailbreak and harmful prompts. Lastly, we further contribute to the field by open-sourcing both the synthetic dataset and the off-topic guardrail models, providing valuable resources for developing guardrails in pre-production environments and supporting future research and development in LLM safety.

A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection

TL;DR

The paper tackles off-topic misuse in LLMs and the limitations of real-world data for guardrails. It introduces a flexible, data-free guardrail development methodology that uses qualitative problem analysis and LLM-driven synthetic data to train both bi-encoder and cross-encoder detectors, producing calibrated probability scores for threshold-based refusals. The approach demonstrates high discrimination (ROC-AUC near 0.99) and strong generalization to jailbreak and harmful prompts, with favorable inference speeds, and is openly released as synthetic data and models to accelerate safety research. Practical deployment considerations, calibration, and limitations are discussed to guide real-world adoption and future work in pre-production safety safeguards.

Abstract

Large Language Models (LLMs) are prone to off-topic misuse, where users may prompt these models to perform tasks beyond their intended scope. Current guardrails, which often rely on curated examples or custom classifiers, suffer from high false-positive rates, limited adaptability, and the impracticality of requiring real-world data that is not available in pre-production. In this paper, we introduce a flexible, data-free guardrail development methodology that addresses these challenges. By thoroughly defining the problem space qualitatively and passing this to an LLM to generate diverse prompts, we construct a synthetic dataset to benchmark and train off-topic guardrails that outperform heuristic approaches. Additionally, by framing the task as classifying whether the user prompt is relevant with respect to the system prompt, our guardrails effectively generalize to other misuse categories, including jailbreak and harmful prompts. Lastly, we further contribute to the field by open-sourcing both the synthetic dataset and the off-topic guardrail models, providing valuable resources for developing guardrails in pre-production environments and supporting future research and development in LLM safety.

Paper Structure

This paper contains 38 sections, 2 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Alignment Challenges in LLMs
  4. Guardrails and Their Gaps
  5. Synthetic Data for LLM Safety
  6. Methodology
  7. Guardrail Development Framework
  8. Step 1: Qualitative Problem Analysis & Edge Case Identification.
  9. Step 2: Synthetic Data Generation via LLM Prompting.
  10. Step 3: Model Training.
  11. Off-Topic Prompt Detection Formulation
  12. Modeling
  13. 1. Fine-Tuned Bi-Encoder Classifier
  14. 2. Fine-Tuned Cross-Encoder Classifier
  15. Experiments and Results
  16. ...and 23 more sections

Figures (4)

  • Figure 1: Example of on- and off-topic user prompts: The goal is to classify whether a user prompt is off-topic with respect to the developer-defined system prompt.
  • Figure 2: Our Guardrail Development Methodology
  • Figure 3: Summary of the two modeling approaches for off-topic prompt detection
  • Figure 4: Calibration Plot on the Synthetic Hold-Out Set. A near-diagonal plot indicates good probability calibration.