Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers

TL;DR

This work addresses the fairness and robustness of closed-source AI Safety Moderation (ASM) classifiers used for social-media moderation and as guardrails for LLM safety. It introduces a unified framework leveraging Demographic Parity ($DP$) and Conditional Statistical Parity ($CSP$) to assess group fairness, and a robustness metric $f^{robust}$ under semantic-preserving perturbations, with $DP = | P(\hat{Y}=1|A=a) - P(\hat{Y}=1|A=a')|$ and $CSP = | P(\hat{Y}=1|A=a, F) - P(\hat{Y}=1|A=a', F) |$. The study empirically evaluates four closed-source ASM services (OpenAI Moderation, Perspective, GCNL, Clarifai) plus an Always Fair baseline on Jigsaw toxicity and Reddit ideology data, finding notable fairness gaps (especially for OpenAI) and robust but brittle behavior under LLM-based perturbations, which can flip unsafe to safe classifications. These findings highlight practical risks in deployment and the need for improved fairness monitoring, robust evaluation, and benchmark perturbations to guide safer moderation and training data curation for LLMs.

Abstract

AI Safety Moderation (ASM) classifiers are designed to moderate content on social media platforms and to serve as guardrails that prevent Large Language Models (LLMs) from being fine-tuned on unsafe inputs. Owing to their potential for disparate impact, it is crucial to ensure that these classifiers: (1) do not unfairly classify content belonging to users from minority groups as unsafe compared to those from majority groups and (2) that their behavior remains robust and consistent across similar inputs. In this work, we thus examine the fairness and robustness of four widely-used, closed-source ASM classifiers: OpenAI Moderation API, Perspective API, Google Cloud Natural Language (GCNL) API, and Clarifai API. We assess fairness using metrics such as demographic parity and conditional statistical parity, comparing their performance against ASM models and a fair-only baseline. Additionally, we analyze robustness by testing the classifiers' sensitivity to small and natural input perturbations. Our findings reveal potential fairness and robustness gaps, highlighting the need to mitigate these issues in future versions of these models.

Figures (4)

• Figure 1: The comparison highlights bias in the OpenAI Moderation API based on the gender aspects of a comment selected from the Jigsaw-Gender dataset ($\checkmark$ indicates Safe and $\times$ indicates Unsafe prediction).
• Figure 2: A small perturbation in the input prompt may convert the ASM classification from Unsafe to Safe. This can be seen in the example above that was inputted to the OpenAI Moderation API ($\checkmark$ indicates Safe and $\times$ indicates Unsafe prediction).
• Figure 3: The demographic parity difference for the four ASM models considered in this work where subfigure A represents OpenAI Moderation API, subfigure B represents Perspective API, subfigure C represents GCNL API, and subfigure D represents Clarifai API. In each subfigure, a lighter background color implies more fairness (i.e. values closer to 0 on both axes). Note that subfigure C (bottom left) is the most fair whereas subfigure A (top left) has significant fairness issues with respect to the Jigsaw-S.O dataset.
• Figure 4: Robustness analysis on all the ASM models considered in this work where subfigure A represents OpenAI Moderation API, subfigure B represents Perspective API, subfigure C represents GCNL API, and subfigure D represents Clarifai API. Here, a cell value represents the portion of inputs that were initially assigned a label shown on the left and have been assigned the label shown at the bottom after the perturbation. For example, the top-left cell in A for the Reddit dataset with value 0.35 implies that 35% of the initially unsafe inputs are still labeled as unsafe after perturbation.