ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models

TL;DR

This work addresses safety gaps in large reasoning models by formalizing CoT moderation with a multi-level risk taxonomy and delivering ReasoningShield, a lightweight yet robust framework. It introduces a two-stage training pipeline (SFT followed by DPO) and a high-quality CoT moderation dataset (ReasoningShield-Train 7K and ReasoningShield-Test 2.2K) to detect risks in intermediate reasoning steps. Empirical results show state-of-the-art performance on CoT moderation, strong generalization to unseen paradigms and data distributions, enhanced explainability through stepwise risk localization, and favorable efficiency on resource-constrained devices. The framework is open-sourced to spur further research and practical adoption in safer LRM deployments.

Abstract

Large Reasoning Models (LRMs) leverage transparent reasoning traces, known as Chain-of-Thoughts (CoTs), to break down complex problems into intermediate steps and derive final answers. However, these reasoning traces introduce unique safety challenges: harmful content can be embedded in intermediate steps even when final answers appear benign. Existing moderation tools, designed to handle generated answers, struggle to effectively detect hidden risks within CoTs. To address these challenges, we introduce ReasoningShield, a lightweight yet robust framework for moderating CoTs in LRMs. Our key contributions include: (1) formalizing the task of CoT moderation with a multi-level taxonomy of 10 risk categories across 3 safety levels, (2) creating the first CoT moderation benchmark which contains 9.2K pairs of queries and reasoning traces, including a 7K-sample training set annotated via a human-AI framework and a rigorously curated 2.2K human-annotated test set, and (3) developing a two-stage training strategy that combines stepwise risk analysis and contrastive learning to enhance robustness. Experiments show that ReasoningShield achieves state-of-the-art performance, outperforming task-specific tools like LlamaGuard-4 by 35.6% and general-purpose commercial models like GPT-4o by 15.8% on benchmarks, while also generalizing effectively across diverse reasoning paradigms, tasks, and unseen scenarios. All resources are released at https://github.com/CosmosYi/ReasoningShield.

Figures (8)

• Figure 1: CoT Moderation vs. Answer Moderation, highlighting the challenges faced by existing moderation models on CoT Moderation: they are often misled by the safe conclusion of reasoning traces but overlook hidden risks in the lengthy intermediate reasoning steps.
• Figure 2: The overall framwork of ReasoningShield. (A) Dataset construction involving reasoning traces generation, human-AI collaborative annotation, and quality control. (B) Two-stage training with SFT on agreed-on samples for structured analysis and DPO on hard negatives for enhanced robustness. (C) Multi-faceted evaluation of ReasoningShield against other moderation models, demonstrating its superior performance in accuracy, generalization, explainability, and efficiency.
• Figure 3: Overview of the ReasoningShield Dataset. (1) Composition: The train split includes 7K samples from four open-source LRMs, while the test split includes 2.2K samples (600 in-domain, 1.6K out-of-domain) from unseen LRMs and datasets. (2) Risk Taxonomy: The pie chart shows category distribution, and the bar chart depicts safety level distribution.
• Figure 4: (Left) Performance on CoT Moderation: ReasoningShield establishes a new SOTA. (Right) Performance on public Answer Moderation datasets: ReasoningShield also achieves superior generalization. Top-3 results are highlighted, and error bars represent 95% confidence intervals.
• Figure 5: Performance of moderation models on CoT Moderation: AIR-Bench (Top-Left), SALAD-Bench (Top-Right), BeaverTails (Bottom-Left), and Jailbreak-Bench (Bottom-Right). The top-three results are highlighted for each dataset, with error bars representing the 95% confidence intervals.