Attacking Open-domain Question Answering by Injecting Misinformation

TL;DR

The paper investigates the vulnerability of open-domain QA systems to misinformation by injecting both human- and model-generated fake documents into the evidential corpus. It introduces BART-FG and Gap Span Filling to generate controllable, realistic misinformation and constructs a large, multilayered pollution framework (including targeted and non-targeted attacks) evaluated across BM25 and dense retrievers with multiple QA models. Key findings show substantial performance degradation across models, with targeted misinformation and scalable neural generation causing the largest drops, and retrievers (especially dense ones) being susceptible due to retrieval of misleading evidence. The work highlights the need for misinformation-aware QA, proposes defense directions (fact-checking integration, quality-aware retrieval, reasoning amidst contradictions), and provides data and code to enable further research in protecting QA systems from misinformation.

Abstract

With a rise in false, inaccurate, and misleading information in propaganda, news, and social media, real-world Question Answering (QA) systems face the challenges of synthesizing and reasoning over misinformation-polluted contexts to derive correct answers. This urgency gives rise to the need to make QA systems robust to misinformation, a topic previously unexplored. We study the risk of misinformation to QA models by investigating the sensitivity of open-domain QA models to corpus pollution with misinformation documents. We curate both human-written and model-generated false documents that we inject into the evidence corpus of QA models and assess the impact on the performance of these systems. Experiments show that QA models are vulnerable to even small amounts of evidence contamination brought by misinformation, with large absolute performance drops on all models. Misinformation attack brings more threat when fake documents are produced at scale by neural models or the attacker targets hacking specific questions of interest. To defend against such a threat, we discuss the necessity of building a misinformation-aware QA system that integrates question-answering and misinformation detection in a joint fashion.

Figures (6)

• Figure 1: Our framework injects human-created and model-generated misinformation documents into the QA evidence repository (left) and evaluates the impact on the performance of open-domain QA systems (right).
• Figure 2: Overview of the BART-FG model, illustrated by an example sentence.
• Figure 3: The EM score for DeBERTa-V3 model with different number of injected fake passages $N$.
• Figure 4: Distribution of error sources when the model is misled by a fake passage and gives a wrong answer.
• Figure 5: An example of human annotation that follows all instructions of the annotation guideline.