Chain-of-Defensive-Thought: Structured Reasoning Elicits Robustness in Large Language Models against Reference Corruption

TL;DR

Chain-of-Defensive-Thought (CoDT) is a prompting-only method that induces a structured defensive reasoning process before answering, aiming to reduce susceptibility to corrupted external references in retrieval-augmented generation. By using few-shot exemplars that demonstrate identifying relevant and reliable contexts and then reasoning with only reliable ones, CoDT substantially boosts robustness across diverse models and tasks (NQ and RealTime QA) against prompt injection and poisoned knowledge, while largely preserving clean performance. The approach is model-agnostic and suitable for open models and black-box APIs, suggesting it could become a strong baseline for systems that rely on external references. Overall, CoDT advances reliability in LLMs handling external information and highlights the value of structured, defensive reasoning in non-reasoning-focused tasks.

Abstract

Chain-of-thought prompting has demonstrated great success in facilitating the reasoning abilities of large language models. In this work, we explore how these enhanced reasoning abilities can be exploited to improve the robustness of large language models in tasks that are not necessarily reasoning-focused. In particular, we show how a wide range of large language models exhibit significantly improved robustness against reference corruption using a simple method called chain-of-defensive-thought, where only a few exemplars with structured and defensive reasoning are provided as demonstrations. Empirically, the improvements can be astounding, especially given the simplicity and applicability of the method. For example, in the Natural Questions task, the accuracy of GPT-4o degrades from 60% to as low as 3% with standard prompting when 1 out of 10 references provided is corrupted with prompt injection attacks. In contrast, GPT-4o using chain-of-defensive-thought prompting maintains an accuracy of 50%.

Figures (5)

• Figure 1: Illustrative exemplars for standard prompting v.s. chain-of-defensive-thought prompting. Chain-of-defensive-thought uses exemplars to prompt models to generate a chain of defensive thought (e.g. 'Reason' highlighted above) before answering.
• Figure 2: Chain-of-defensive-thought unlocks the robustness in a wide range of large language models against reference corruption. Here the robustness metric is the average robust accuracy over two benchmarks where for each benchmark the minimum accuracy obtained across attack types is accounted. Please refer to section \ref{['sec:eval']} for evaluation details.
• Figure 3: Comparing the clean performance of standard prompting v.s. chain-of-defensive-thought. In most cases, there is neither considerable increase nor decrease regarding clean performance (i.e., the performance with no external reference corrupted) of language models when using chain-of-defensive-thought to improve their robustness.
• Figure 4: Accuracy and attack success rate for various models against prompt injection attacks PIA.
• Figure 5: Accuracy and attack success rate for various models against knowledge corruption attacks poisonedrag.