Thought Purity: A Defense Framework For Chain-of-Thought Attack

TL;DR

This work addresses security vulnerabilities in chain-of-thought reasoning within LRMs by proposing Thought Purity (TP), a defense framework that combines a safety-aware data pipeline with GRPO-based RL and adaptive monitoring. TP uses explicit safety signaling through tags like <suspect> and <harm> and a three-way defense metric to counter Chain-of-Thought Attacks without sacrificing reasoning efficacy. Empirical results across multiple datasets and model families show TP improves Cure Rate and Reject Rate while reducing Attack Success, and analyze how model characteristics influence vulnerability and defense effectiveness. The approach advances the security-functionality trade-off for next-generation AI systems by integrating backdoor defenses into the RL training loop and enabling scalable, data-driven defenses against CoTA.

Abstract

While reinforcement learning-trained Large Reasoning Models (LRMs, e.g., Deepseek-R1) demonstrate advanced reasoning capabilities in the evolving Large Language Models (LLMs) domain, their susceptibility to security threats remains a critical vulnerability. This weakness is particularly evident in Chain-of-Thought (CoT) generation processes, where adversarial methods like backdoor prompt attacks can systematically subvert the model's core reasoning mechanisms. The emerging Chain-of-Thought Attack (CoTA) reveals this vulnerability through exploiting prompt controllability, simultaneously degrading both CoT safety and task performance with low-cost interventions. To address this compounded security-performance vulnerability, we propose Thought Purity (TP): a defense framework that systematically strengthens resistance to malicious content while preserving operational efficacy. Our solution achieves this through three synergistic components: (1) a safety-optimized data processing pipeline (2) reinforcement learning-enhanced rule constraints (3) adaptive monitoring metrics. Our approach establishes the first comprehensive defense mechanism against CoTA vulnerabilities in reinforcement learning-aligned reasoning systems, significantly advancing the security-functionality equilibrium for next-generation AI architectures.

Figures (4)

• Figure 1: Backdoor attackers provide triggers and redundant reasoning mapping examples by injecting system prompts, and then activate it by adding triggers in user prompts.
• Figure 2: This figure introduces common attack patterns of CoTA and briefly describes the data processing Pipeline. The right part roughly introduces the method of implementing the defense process template under the guidance of the TP framework.
• Figure 3: The distribution of the model performance under different RL Settings. In a large number of repetitive experiments, RL has demonstrated its unstable characteristics. Even minor parameter changes can lead to deviations in the results. However, in terms of distribution, the three different RL Settings generally have a distinct tendency, roughly as shown in the figure.
• Figure 4: Above, regarding the performance of common LLMs under the BadChain attack. In the lower left corner, the convergence of rewards during GRPO training. In the lower right corner, the performance changes of LRMs of different scales under the reinforcement learning effect of the TP paradigm (Taking the Qwen3 series +GSM8K as an example).