Unsupervised dense retrieval with conterfactual contrastive learning

TL;DR

This work tackles the vulnerability and explainability gaps in dense retrieval by introducing counterfactual contrastive learning guided by Shapley-value-based key passage extraction. It formalizes a retrieval framework where relevance signals are dissected at the passage level, and key passages are identified via a model-agnostic counterfactual analysis; deleting or altering these passages yields measurable shifts in relevance, enabling targeted regularization. The authors propose a composite loss L = L_{cla} + \alpha (L_{neg} + L_{adv}) + \beta L_{pos}, integrating classical contrastive objectives with counterfactual and adversarial components to train models that are both accurate and robust to adversarial manipulation. Experimental results on MSMARCO demonstrate improved key-passage extraction and enhanced resistance to diverse attacks, without sacrificing overall document retrieval performance. The approach offers practical benefits for interpretability and security in large-scale IR systems, with implications for pretraining tasks and deployment in sensitive search contexts.

Abstract

Efficiently retrieving a concise set of candidates from a large document corpus remains a pivotal challenge in Information Retrieval (IR). Neural retrieval models, particularly dense retrieval models built with transformers and pretrained language models, have been popular due to their superior performance. However, criticisms have also been raised on their lack of explainability and vulnerability to adversarial attacks. In response to these challenges, we propose to improve the robustness of dense retrieval models by enhancing their sensitivity of fine-graned relevance signals. A model achieving sensitivity in this context should exhibit high variances when documents' key passages determining their relevance to queries have been modified, while maintaining low variances for other changes in irrelevant passages. This sensitivity allows a dense retrieval model to produce robust results with respect to attacks that try to promote documents without actually increasing their relevance. It also makes it possible to analyze which part of a document is actually relevant to a query, and thus improve the explainability of the retrieval model. Motivated by causality and counterfactual analysis, we propose a series of counterfactual regularization methods based on game theory and unsupervised learning with counterfactual passages. Experiments show that, our method can extract key passages without reliance on the passage-level relevance annotations. Moreover, the regularized dense retrieval models exhibit heightened robustness against adversarial attacks, surpassing the state-of-the-art anti-attack methods.

Figures (1)

• Figure 1: Illustration of the counterfactual and adversarial documents as pivots in the embedding space.