Evaluating Robustness of Generative Search Engine on Adversarial Factual Questions

TL;DR

This paper investigates the adversarial robustness of generative search engines under black-box access by crafting seven factual attack methods and applying them to a Wikipedia-derived corpus, generating 534 attack sentences in both declarative and interrogative forms. It evaluates multiple engines (Bing Copilot, PerplexityAI, YouChat) and baselines (Gemini, GPT-3.5, GPT-4) across six metrics, finding that retrieval-augmented systems are generally more susceptible to factual errors than non-retrieval LLMs, with notable variations across attack types and modes. The analysis reveals contextual contradictions, high citation precision alongside rising ASR, and gaps in numerical reasoning, as well as the greater effectiveness of temporal/distraction and multihop-based attacks. The results emphasize safety risks and the need for thorough evaluation and mitigation strategies before deploying these systems, highlighting the importance of robust verification, careful prompt design, and consideration of input form and knowledge integration in defense planning.

Abstract

Generative search engines have the potential to transform how people seek information online, but generated responses from existing large language models (LLMs)-backed generative search engines may not always be accurate. Nonetheless, retrieval-augmented generation exacerbates safety concerns, since adversaries may successfully evade the entire system by subtly manipulating the most vulnerable part of a claim. To this end, we propose evaluating the robustness of generative search engines in the realistic and high-risk setting, where adversaries have only black-box system access and seek to deceive the model into returning incorrect responses. Through a comprehensive human evaluation of various generative search engines, such as Bing Chat, PerplexityAI, and YouChat across diverse queries, we demonstrate the effectiveness of adversarial factual questions in inducing incorrect responses. Moreover, retrieval-augmented generation exhibits a higher susceptibility to factual errors compared to LLMs without retrieval. These findings highlight the potential security risks of these systems and emphasize the need for rigorous evaluation before deployment.

Figures (7)

• Figure 1: Explanation of seven different attack methods.
• Figure 2: The generative search engine provided answers with conflicting contexts.
• Figure 3: An example of calculating the Citation Precise in sentences related to adversarial attacks.
• Figure 4: The change in the Citation Precise of the search engine after removing irrelevant references.
• Figure 5: The impact of attack words with different grammatical importance and entity types on ASR.