Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models

TL;DR

This paper uncovers a novel vulnerability in reasoning LLMs: query-agnostic triggers that do not alter problem semantics can dramatically mislead models. It introduces CatAttack, an automated, three-stage jailbreak pipeline using a cheaper proxy model to discover universal triggers, which then transfer to stronger targets and across model families. The study demonstrates substantial increases in incorrect outputs and longer response lengths, with transferable effects observed from DeepSeek V3 to DeepSeek R1 and beyond to Qwen, Llama-3.1, and Mistral, underscoring widespread security and efficiency concerns and prompting exploration of defenses such as targeted fine-tuning and instruction-based filtering.

Abstract

We investigate the robustness of reasoning models trained for step-by-step problem solving by introducing query-agnostic adversarial triggers - short, irrelevant text that, when appended to math problems, systematically mislead models to output incorrect answers without altering the problem's semantics. We propose CatAttack, an automated iterative attack pipeline for generating triggers on a weaker, less expensive proxy model (DeepSeek V3) and successfully transfer them to more advanced reasoning target models like DeepSeek R1 and DeepSeek R1-distilled-Qwen-32B, resulting in greater than 300% increase in the likelihood of the target model generating an incorrect answer. For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong. Our findings highlight critical vulnerabilities in reasoning models, revealing that even state-of-the-art models remain susceptible to subtle adversarial inputs, raising security and reliability concerns. The CatAttack triggers dataset with model responses is available at https://huggingface.co/datasets/collinear-ai/cat-attack-adversarial-triggers.

Figures (7)

• Figure 1: The iterative approach of CatAttack, wherein the attacker model, the proxy target model, and the judge model iteratively generate successful jailbreaks on math problems. The attacker sends a candidate math prompt $p$ to the proxy target which then solves the problem and generates a solution response $r'$, the judge then generates a score $s$ based on whether the final answer matches the ground truth or not. If the answer matches, the judge outputs no for ' is jailbroken?' and sends feedback to the attacker model. If the answer does not match, it means the attack worked and the iteration breaks for that prompt. The attacks that pass are then assessed on the actual target model.
• Figure 2: Relative increase in error rates after CatAttack by source. This figure shows the multiplicative increase in error rates for Deepseek-R1 and DeepSeek R1-distill-Qwen-32B models across different problem sources following the suffix attack. Sources are grouped into three difficulty categories: easy (left), medium (center), and hard (right), separated by vertical dotted lines, with no particular ordering within each category. Values for cn_k12 and gsm8k for Deepseek-R1 are capped at 15x for visualization purposes, where error rates increased from $0\%$ to non-zero levels. This capping preserves the readability of the chart while still indicating the substantial impact in these cases. We note that there were no successful errors for amc_aime.
• Figure 3: Response length analysis before and after CatAttack. Scatter plots compare token lengths of responses to original prompts (x-axis) versus modified prompts (y-axis) for Deepseek-R1 (right) and DeepSeek R1-distill-Qwen-32B (left) models. Logarithmic scales are used on both axes to accommodate the wide range of response lengths. Diagonal lines represent different multipliers of the original length (1x, 1.5x, 2x, etc.). Points above the 1x line represent responses that increased in length after modification, with higher lines indicating greater increases.
• Figure 4: Example of tokens generated by the DeepSeek R1 for the original prompt and the adversarial prompt. All reasoning tokens are truncated due to brevity.
• Figure 5: Example of tokens generated by the DeepSeek R1 for the original prompt and the adversarial prompt. All reasoning tokens are truncated due to brevity.