Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systems
Hongru Song, Yu-an Liu, Ruqing Zhang, Jiafeng Guo, Yixing Fan
TL;DR
The paper investigates vulnerabilities of retrieval-augmented generation (RAG) systems that use deep reasoning LLMs. It introduces chain-of-thought poisoning, which extracts a reasoning-template from a victim RAG and generates adversarial documents with fabricated reasoning chains to mislead the model. Experiments on the MS MARCO dataset show the proposed method improves attack success on R1-based RAG relative to existing knowledge-base poisoning methods, and human annotators rate the naturalness of the adversarial reasoning. The work highlights that deep reasoning both strengthens RAG performance and opens new attack surfaces, motivating future defense research against reasoning-chain level attacks.
Abstract
Retrieval-augmented generation (RAG) systems can effectively mitigate the hallucination problem of large language models (LLMs),but they also possess inherent vulnerabilities. Identifying these weaknesses before the large-scale real-world deployment of RAG systems is of great importance, as it lays the foundation for building more secure and robust RAG systems in the future. Existing adversarial attack methods typically exploit knowledge base poisoning to probe the vulnerabilities of RAG systems, which can effectively deceive standard RAG models. However, with the rapid advancement of deep reasoning capabilities in modern LLMs, previous approaches that merely inject incorrect knowledge are inadequate when attacking RAG systems equipped with deep reasoning abilities. Inspired by the deep thinking capabilities of LLMs, this paper extracts reasoning process templates from R1-based RAG systems, uses these templates to wrap erroneous knowledge into adversarial documents, and injects them into the knowledge base to attack RAG systems. The key idea of our approach is that adversarial documents, by simulating the chain-of-thought patterns aligned with the model's training signals, may be misinterpreted by the model as authentic historical reasoning processes, thus increasing their likelihood of being referenced. Experiments conducted on the MS MARCO passage ranking dataset demonstrate the effectiveness of our proposed method.