Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Taxonomy-Adaptive Moderation Model with Robust Guardrails for Large Language Models

Mahesh Kumar Nandwana, Youngwan Lim, Joseph Liu, Alex Yang, Varun Notibala, Nishchaie Khanna

TL;DR

The paper tackles the challenge of safe LLM deployment across diverse contexts by introducing taxonomy-adaptive guardrails. It presents Roblox Guard 1.0, a LoRA-fine-tuned Llama-3.1-8B-Instruct model trained on a 384k-scale open/synthetic safety dataset, augmented with chain-of-thought rationales and input inversion to improve generalization to unseen safety categories. A new evaluation suite, RobloxGuard-Eval, provides fine-grained, platform-aligned taxonomy coverage across 25 categories and end-to-end prompt/response assessment. Empirical results show state-of-the-art or competitive performance on a wide range of benchmarks, including strong robustness to taxonomy drift, and the ablations quantify the contributions of synthetic data, CoT, and input inversion to overall performance.

Abstract

Large Language Models (LLMs) are typically aligned for safety during the post-training phase; however, they may still generate inappropriate outputs that could potentially pose risks to users. This challenge underscores the need for robust safeguards that operate across both model inputs and outputs. In this work, we introduce Roblox Guard 1.0, a state-of-the-art instruction fine-tuned LLM designed to enhance the safety of LLM systems through comprehensive input-output moderation, using a pipeline of LLMs to enhance moderation capability. Built on the Llama-3.1-8B-Instruct backbone, our model is instruction fine-tuned to generalize across previously unseen safety taxonomies and demonstrates strong performance on out-of-domain safety benchmarks. The instruction fine-tuning process uses a mix of synthetic and open-source safety datasets, augmented with chain-of-thought (CoT) rationales and input inversion to enhance contextual understanding and decision making. To support systematic evaluation, we also release RobloxGuard-Eval, a new benchmark featuring an extensible safety taxonomy to assess the effectiveness of LLM guardrails and moderation frameworks.

Taxonomy-Adaptive Moderation Model with Robust Guardrails for Large Language Models

TL;DR

The paper tackles the challenge of safe LLM deployment across diverse contexts by introducing taxonomy-adaptive guardrails. It presents Roblox Guard 1.0, a LoRA-fine-tuned Llama-3.1-8B-Instruct model trained on a 384k-scale open/synthetic safety dataset, augmented with chain-of-thought rationales and input inversion to improve generalization to unseen safety categories. A new evaluation suite, RobloxGuard-Eval, provides fine-grained, platform-aligned taxonomy coverage across 25 categories and end-to-end prompt/response assessment. Empirical results show state-of-the-art or competitive performance on a wide range of benchmarks, including strong robustness to taxonomy drift, and the ablations quantify the contributions of synthetic data, CoT, and input inversion to overall performance.

Abstract

Large Language Models (LLMs) are typically aligned for safety during the post-training phase; however, they may still generate inappropriate outputs that could potentially pose risks to users. This challenge underscores the need for robust safeguards that operate across both model inputs and outputs. In this work, we introduce Roblox Guard 1.0, a state-of-the-art instruction fine-tuned LLM designed to enhance the safety of LLM systems through comprehensive input-output moderation, using a pipeline of LLMs to enhance moderation capability. Built on the Llama-3.1-8B-Instruct backbone, our model is instruction fine-tuned to generalize across previously unseen safety taxonomies and demonstrates strong performance on out-of-domain safety benchmarks. The instruction fine-tuning process uses a mix of synthetic and open-source safety datasets, augmented with chain-of-thought (CoT) rationales and input inversion to enhance contextual understanding and decision making. To support systematic evaluation, we also release RobloxGuard-Eval, a new benchmark featuring an extensible safety taxonomy to assess the effectiveness of LLM guardrails and moderation frameworks.

Paper Structure

This paper contains 17 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Guardrail Models
  4. LLM Safety Benchmark Datasets
  5. Taxonomy for LLM Safety
  6. Datasets
  7. Multi-Stage Synthetic Data Generation
  8. RobloxGuard Evaluation Set
  9. Roblox Guard 1.0 Training Dataset
  10. Chain-of-Thought
  11. Input Inversion
  12. Benchmarking Datasets
  13. Experiments
  14. Roblox Guard 1.0 Training
  15. Results
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Synthetic data generation pipeline for Roblox Guard 1.0
  • Figure 2: Input Inversion. We permute the ordering of target components (Chain-of-Thought, Label, and Category) during training. This prevents the model from overfitting to a specific output format and improves robustness, allowing RobloxGuard to be taxonomy-adaptive.
  • Figure 3: Overview of the three-stage synthetic data pipeline. (a) Stage 1 (Generation): An AI security tester prompt generates an adversarial scenario (System Prompt, User Message). (b) Stage 2 (Response): The scenario is used to prompt a target LLM, producing an AI Output. (c) Stage 3 (Validation): A 'judge' LLM, guided by Community Standards, evaluates the output and produces a final binary violation label.