Scaling Reinforcement Learning for Content Moderation with Large Language Models

TL;DR

Scaling reinforcement learning for content moderation with large language models addresses label scarcity and evolving policies by empirically evaluating diverse RL training recipes, reward shaping, and evaluation frameworks across three real-world tasks. The study introduces a practical RL pipeline with GRPO optimization, verifiable rewards, rubric-based reasoning, Monte Carlo score aggregation, and reflection-aided prompting to achieve policy-aligned moderation. Key findings show sigmoid-like scaling with data, rollouts, and compute, and up to 100x data efficiency over supervised fine-tuning, especially on tasks requiring complex reasoning. The work offers actionable design guidelines to stabilize training, calibrate confidence, and deploy industrial-scale moderation systems that adapt to changing policy requirements.

Abstract

Content moderation at scale remains one of the most pressing challenges in today's digital ecosystem, where billions of user- and AI-generated artifacts must be continuously evaluated for policy violations. Although recent advances in large language models (LLMs) have demonstrated strong potential for policy-grounded moderation, the practical challenges of training these systems to achieve expert-level accuracy in real-world settings remain largely unexplored, particularly in regimes characterized by label sparsity, evolving policy definitions, and the need for nuanced reasoning beyond shallow pattern matching. In this work, we present a comprehensive empirical investigation of scaling reinforcement learning (RL) for content classification, systematically evaluating multiple RL training recipes and reward-shaping strategies-including verifiable rewards and LLM-as-judge frameworks-to transform general-purpose language models into specialized, policy-aligned classifiers across three real-world content moderation tasks. Our findings provide actionable insights for industrial-scale moderation systems, demonstrating that RL exhibits sigmoid-like scaling behavior in which performance improves smoothly with increased training data, rollouts, and optimization steps before gradually saturating. Moreover, we show that RL substantially improves performance on tasks requiring complex policy-grounded reasoning while achieving up to 100x higher data efficiency than supervised fine-tuning, making it particularly effective in domains where expert annotations are scarce or costly.

Figures (8)

• Figure 1: Accuracy-based rewards induce reward hacking: explanation length collapses over training, and responses degenerate into short label assertions.
• Figure 2: Faithfulness and factuality under RL ablations. “Instruction hallucination” and “factuality hallucination” are the complements of faithfulness and factuality, respectively.
• Figure 3: Monte-Carlo Sampling at Different N (number of rollouts) and T (temperature) for Task1 and Task2
• Figure 4: Comparison of label token probability distribution: without Monte-Carlo sampling vs with Monte-Carlo sampling
• Figure 5: Comparison of score/probability distributions for last- vs. first-decision scoring in reflection-aided prompting strategy.