Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition

TL;DR

This study systematically reevaluates the claim that dense retrievers rely primarily on pre-training knowledge, rather than fine-tuning, for retrieval performance. It extends prior work by testing multiple backbones (BERT, LLaMA), embedding representations (CLS, mean pooling, EOS), and datasets (Natural Questions, MS MARCO), using linear probing and integrated gradients to analyze knowledge distribution. Results show that pre-trained knowledge underpins retrieval for some configurations (e.g., DPR with CLS), while fine-tuning can reallocate neuron activations; however, this pattern does not generalize across mean-pooled or decoder-based models, where different retrieval dynamics emerge. The findings highlight architecture- and pooling-dependent effects on knowledge flow in dense retrievers and underscore the need for broader, mechanistic analyses across diverse architectures. The work also emphasizes reproducibility by providing public code and datasets to support future investigations.

Abstract

Dense retrievers utilize pre-trained backbone language models (e.g., BERT, LLaMA) that are fine-tuned via contrastive learning to perform the task of encoding text into sense representations that can be then compared via a shallow similarity operation, e.g. inner product. Recent research has questioned the role of fine-tuning vs. that of pre-training within dense retrievers, specifically arguing that retrieval knowledge is primarily gained during pre-training, meaning knowledge not acquired during pre-training cannot be sub-sequentially acquired via fine-tuning. We revisit this idea here as the claim was only studied in the context of a BERT-based encoder using DPR as representative dense retriever. We extend the previous analysis by testing other representation approaches (comparing the use of CLS tokens with that of mean pooling), backbone architectures (encoder-only BERT vs. decoder-only LLaMA), and additional datasets (MSMARCO in addition to Natural Questions). Our study confirms that in DPR tuning, pre-trained knowledge underpins retrieval performance, with fine-tuning primarily adjusting neuron activation rather than reorganizing knowledge. However, this pattern does not hold universally, such as in mean-pooled (Contriever) and decoder-based (LLaMA) models. We ensure full reproducibility and make our implementation publicly available at https://github.com/ielab/DenseRetriever-Knowledge-Acquisition.

Figures (8)

• Figure 1: Results for linear probing experiments on the NQ dataset. The results are based on four different configurations for the number of passages in the probe ($N=2,3,4,5$), each represented by a sub-plot. The x-axis indicates the number of layers used for embedding learning. A two-tailed t-test ($p<0.05$) with Bonferroni correction between each DPR model and the BERT-base-uncased backbone model is marked with *; reichman2024dense did not perform statistical significance analysis so this aspect cannot be cross-checked.
• Figure 2: Generalization of linear probing experiments to MS MARCO. The results are based on four different configurations for the number of passages in the probe ($N=2,3,4,5$), each represented by a sub-plot. The x-axis indicates the number of layers used for embedding learning. A two-tailed t-test ($p<0.05$) with Bonferroni correction between each DPR model with bert-based-unacsed backbone model is marked with *.
• Figure 3: Generalization of Linear Probing experiment when different mean polling strategy is used (Contriever). The results are based on four different configurations for the number of passages in the probe ($N=2,3,4,5$), each represented by a sub-plot. The x-axis indicates the number of layers used for embedding learning. A two-tailed t-test ($p<0.05$) with Bonferroni correction between each DPR model with bert-based-unacsed backbone model is marked with *.
• Figure 4: Generalization of Linear Probing Accuracy when decoder model backbone is used (Llama), EOS tokens are used for embedding representation. The results are based on four different configurations for the number of passages in the probe ($N=2,3,4,5$), each represented by a sub-plot. The x-axis indicates the number of layers used for embedding learning. A two-tailed t-test ($p<0.05$) with Bonferroni correction between each DPR model with bert-based-unacsed backbone model is marked with *.
• Figure 5: Results for knowledge neuron activation comparisons obtained on the NQ dataset. The plots compare DPR-query encoder vs. BERT-CLS using questions; and DPR-passage vs. BERT-CLS backbone model using positive passages.