Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric

TL;DR

LATTE introduces a structured toxicity-investigation framework and a zero-shot toxicity evaluator that uses qualified LLMs to adapt to context-specific toxicity definitions without retraining. By decomposing toxicity into Demeaning, Partiality, and Ethical Preference, and by identifying safe domains through neutrality checks, LATTE achieves superior F1 and accuracy on multiple datasets compared with traditional detectors. The approach highlights the influence of upstream neutrality and prompt design on evaluation reliability and demonstrates robustness to definition shifts and perturbations, though it acknowledges limitations in ethical-preference assessment and computational cost. Overall, LATTE provides a practical, adaptable tool for safety-focused toxicity evaluation in diverse LLM applications.

Abstract

In the pursuit of developing Large Language Models (LLMs) that adhere to societal standards, it is imperative to detect the toxicity in the generated text. The majority of existing toxicity metrics rely on encoder models trained on specific toxicity datasets, which are susceptible to out-of-distribution (OOD) problems and depend on the dataset's definition of toxicity. In this paper, we introduce a robust metric grounded on LLMs to flexibly measure toxicity according to the given definition. We first analyze the toxicity factors, followed by an examination of the intrinsic toxic attributes of LLMs to ascertain their suitability as evaluators. Finally, we evaluate the performance of our metric with detailed analysis. Our empirical results demonstrate outstanding performance in measuring toxicity within verified factors, improving on conventional metrics by 12 points in the F1 score. Our findings also indicate that upstream toxicity significantly influences downstream metrics, suggesting that LLMs are unsuitable for toxicity evaluations within unverified factors.

Figures (25)

• Figure 1: An example of LATTE's toxicity detection results compared to traditional detectors. The blue box represents metrics that are trained on each criteria or dataset, whereas LATTE is a metric that detects toxicity using only the toxicity definition without any further training procedure.
• Figure 2: Through the qualification process, we filter out the biased LLMs. Next, we adopt qualified LLMs to our proposed metric LLMs As ToxiciTy Evaluator (LATTE) for guaranteed factors.
• Figure 3: Awareness of the Demeaning Factor Test
• Figure 4: LATTE Prompt Content Test: GPT-4
• Figure 5: Comparison on Detecting Toxicity in Prosocial Dialogue