MultiContrievers: Analysis of Dense Retrieval Representations

TL;DR

It is found that contriever models have significantly increased extractability, but extractability usually correlates poorly with benchmark performance, and there is high sensitivity to both random initialisation and to data shuffle, suggesting that future retrieval research should test across a wider spread of both.

Abstract

Dense retrievers compress source documents into (possibly lossy) vector representations, yet there is little analysis of what information is lost versus preserved, and how it affects downstream tasks. We conduct the first analysis of the information captured by dense retrievers compared to the language models they are based on (e.g., BERT versus Contriever). We use 25 MultiBert checkpoints as randomized initialisations to train MultiContrievers, a set of 25 contriever models. We test whether specific pieces of information -- such as gender and occupation -- can be extracted from contriever vectors of wikipedia-like documents. We measure this extractability via information theoretic probing. We then examine the relationship of extractability to performance and gender bias, as well as the sensitivity of these results to many random initialisations and data shuffles. We find that (1) contriever models have significantly increased extractability, but extractability usually correlates poorly with benchmark performance 2) gender bias is present, but is not caused by the contriever representations 3) there is high sensitivity to both random initialisation and to data shuffle, suggesting that future retrieval research should test across a wider spread of both.

Figures (11)

• Figure 1: Part 1: We train 25 Contrievers from the 25 MultiBerts, and compare the information theoretic extractability of gender and occupation from each of their representations of documents. Part 2: We then compare these to metrics of performance and of gender bias to better understand the properties of dense retrievers.
• Figure 2: Bert and Contriever compression for gender and occupation over all seeds. Y-axes have different scales (gender is much larger); higher numbers mean more extractability and more regular representations. Contriever has more uniform compression across seeds, and a lower ratio of gender:occupation, which means less shortcutting.
• Figure 3: Scatterplots of the correlation between x-axis compression (ratio of uniform to online codelength) and y-axis performance (NDCG@10), for different datasets (NQ, MSMARCO) at left and entity subsets of NQ at right. Colours are different seeds, and are held constant across graphs.
• Figure 4: Ranking of best performing seed per dataset (one colour per seed). For legibility, NDCG@10 values are scaled, and all seeds with middle performance are not pictured (10 included). One seed is arbitrarily given a star marker to aid visual interpretation.
• Figure 5: Distribution of performance (NDCG@10) for the 24 MultiContrievers, per BEIR dataset, performance on x-axis, number of models with that value on y-axis. Dashed line indicates reference performance from previous work. While for some datasets the reference performance sits at or near the mean of the MultiContriever distribution, for some the reference performance is an outlier. There is sufficiently high variance that performance improvements from random seed can exceed those from continuing to train on supervised fine-tuning data.