Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment

TL;DR

This work systematically explore various tuning methods by leveraging a small set of training data to adapt these models as proxy defense mechanisms to detect malicious inputs and provide a reasoning for their verdicts, thereby preventing the exploitation of conversational agents.

Abstract

Large Language Models (LLMs) have demonstrated powerful capabilities that render them valuable in different applications, including conversational AI products. It is paramount to ensure the security and reliability of these products by mitigating their vulnerabilities towards malicious user interactions, which can lead to the exposure of great risks and reputational repercussions. In this work, we present a comprehensive study on the efficacy of fine-tuning and aligning Chain-of-Thought (CoT) responses of different LLMs that serve as input moderation guardrails. We systematically explore various tuning methods by leveraging a small set of training data to adapt these models as proxy defense mechanisms to detect malicious inputs and provide a reasoning for their verdicts, thereby preventing the exploitation of conversational agents. We rigorously evaluate the efficacy and robustness of different tuning strategies to generalize across diverse adversarial and malicious query types. Our experimental results outline the potential of alignment processes tailored to a varied range of harmful input queries, even with constrained data resources. These techniques significantly enhance the safety of conversational AI systems and provide a feasible framework for deploying more secure and trustworthy AI-driven interactions.

Figures (9)

• Figure 1: An example interface of the Input Moderation Guardrail as proxy defense.
• Figure 2: Example of two "correct" Input Guardrail responses to a fraudulent user query, with the top CoT explanation not aligning with our definition of a concise and helpful explanation, in contrast to one in the bottom.
• Figure 3: F1, ADR, FPR, and Invalid Response Ratio based on evaluating four base LLMs, and those tuned by SFT, SFT + DPO, and SFT + KTO.
• Figure 4: ADR for jailbreak prompts, malicious queries with jailbreak prompts, and stand-alone malicious queries, across the different LLMs and tuning techniques.
• Figure 5: F1, Recall, False Positive Rate (FPR), and invalid prediction ratio based on evaluating each of the four base LLMs using the prompt including and excluding the CoT instruction, respectively.