Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval

João Coelho, Bruno Martins, João Magalhães, Jamie Callan, Chenyan Xiong

TL;DR

Experiments with the MS-MARCO document collection reveal that after contrastive pre-training the model already generates embeddings that better capture early contents of the input, with fine-tuning further aggravating this effect.

Abstract

This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval. We build on previous research that demonstrated loss of information in the middle of input sequences for causal language models, extending it to the domain of representation learning. We examine positional biases at various stages of training for an encoder-decoder model, including language model pre-training, contrastive pre-training, and contrastive fine-tuning. Experiments with the MS-MARCO document collection reveal that after contrastive pre-training the model already generates embeddings that better capture early contents of the input, with fine-tuning further aggravating this effect.

Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval

TL;DR

Experiments with the MS-MARCO document collection reveal that after contrastive pre-training the model already generates embeddings that better capture early contents of the input, with fine-tuning further aggravating this effect.

Abstract

This study investigates the existence of positional biases in Transformer-based models for text representation learning, particularly in the context of web document retrieval. We build on previous research that demonstrated loss of information in the middle of input sequences for causal language models, extending it to the domain of representation learning. We examine positional biases at various stages of training for an encoder-decoder model, including language model pre-training, contrastive pre-training, and contrastive fine-tuning. Experiments with the MS-MARCO document collection reveal that after contrastive pre-training the model already generates embeddings that better capture early contents of the input, with fine-tuning further aggravating this effect.
Paper Structure (17 sections, 1 equation, 5 figures, 5 tables)

This paper contains 17 sections, 1 equation, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Model Architecture
  5. Dense Retriever Training Pipeline
  6. Experiments
  7. Retrieval Performance
  8. Impact of Relevant Passage Position
  9. Contrastive Pre-training Location Bias
  10. Span Corruption Location Bias
  11. Conclusions and Future Work
  12. Training Details
  13. Hyperparameters
  14. Span Corruption Pre-training
  15. Contrastive Pre-training
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Performance of T5-2K and RepLLaMA. Full lines represent the unchanged version of the documents. Dashed lines represent the variations obtained when the relevant passages are moved to a different position.
  • Figure 2: Distribution for the starting position (characters) of relevant passages within 75,000 documents from the MS-MARCO training split.
  • Figure 3: Cosine similarity distribution for exact matching of sub-strings in different locations, using a sample of 24,000 MS-MARCO documents, for the T5-2K model after contrastive pre-training using both decoder-pooling and average-pooling.
  • Figure 4: Span prediction accuracy on different zones of the input, using 7000 random 3-token spans per window.
  • Figure 5: Cosine similarity distribution for exact matching of sub-strings in different locations, using a sample of 24,000 MS-MARCO documents, for the T5-2K model after language model pre-training.