Process or Result? Manipulated Ending Tokens Can Mislead Reasoning LLMs to Ignore the Correct Reasoning Steps

TL;DR

The paper investigates a vulnerability, Compromising Thought (CPT), where subtle tampering of ending tokens in reasoning traces can mislead reasoning LLMs to accept incorrect results. It formalizes CPT, introduces a quantitative resistance framework with r_CPT and R_CPT, and evaluates three prompting interventions to bolster CPT resistance across multiple models and arithmetic/word-problem tasks. Key findings show that local ending-token manipulations can have greater impact on reasoning than structural changes, and that even strong reasoning models like DeepSeek-R1 can exhibit a thinking-stopped failure under CPT in certain contexts. The work highlights important security and robustness considerations for reasoning-intensive applications relying on long Chain-of-Thought, and suggests practical prompting strategies to mitigate CPT risks.

Abstract

Recent reasoning large language models (LLMs) have demonstrated remarkable improvements in mathematical reasoning capabilities through long Chain-of-Thought. The reasoning tokens of these models enable self-correction within reasoning chains, enhancing robustness. This motivates our exploration: how vulnerable are reasoning LLMs to subtle errors in their input reasoning chains? We introduce "Compromising Thought" (CPT), a vulnerability where models presented with reasoning tokens containing manipulated calculation results tend to ignore correct reasoning steps and adopt incorrect results instead. Through systematic evaluation across multiple reasoning LLMs, we design three increasingly explicit prompting methods to measure CPT resistance, revealing that models struggle significantly to identify and correct these manipulations. Notably, contrary to existing research suggesting structural alterations affect model performance more than content modifications, we find that local ending token manipulations have greater impact on reasoning outcomes than structural changes. Moreover, we discover a security vulnerability in DeepSeek-R1 where tampered reasoning tokens can trigger complete reasoning cessation. Our work enhances understanding of reasoning robustness and highlights security considerations for reasoning-intensive applications.

Figures (9)

• Figure 1: CPT overview. In the original reasoning tokens, the correct ending tokens 159,973,388,714,262 will be altered to the incorrect answer 159,973,389,714,262.
• Figure 2: Overview of reasoning tokens for solving mathematical problems and specific objectives of tampering.
• Figure 3: The approaches of exploring the resistance of reasoning LLMs to CPT.
• Figure 4: Evaluation result of CPT towards reasoning LLMs in standalone arithmetic tasks.
• Figure 5: Comparative evaluation of reasoning LLMs on addition and multiplication in CPT.