Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SoftQE: Learned Representations of Queries Expanded by LLMs

Varad Pimpalkhute, John Heyer, Xusen Yin, Sameer Gupta

TL;DR

While improvements over various strong baselines on in-domain MS-MARCO metrics are marginal, SoftQE improves performance by 2.83 absolute percentage points on average on five out-of-domain BEIR tasks.

Abstract

We investigate the integration of Large Language Models (LLMs) into query encoders to improve dense retrieval without increasing latency and cost, by circumventing the dependency on LLMs at inference time. SoftQE incorporates knowledge from LLMs by mapping embeddings of input queries to those of the LLM-expanded queries. While improvements over various strong baselines on in-domain MS-MARCO metrics are marginal, SoftQE improves performance by 2.83 absolute percentage points on average on five out-of-domain BEIR tasks.

SoftQE: Learned Representations of Queries Expanded by LLMs

TL;DR

While improvements over various strong baselines on in-domain MS-MARCO metrics are marginal, SoftQE improves performance by 2.83 absolute percentage points on average on five out-of-domain BEIR tasks.

Abstract

We investigate the integration of Large Language Models (LLMs) into query encoders to improve dense retrieval without increasing latency and cost, by circumventing the dependency on LLMs at inference time. SoftQE incorporates knowledge from LLMs by mapping embeddings of input queries to those of the LLM-expanded queries. While improvements over various strong baselines on in-domain MS-MARCO metrics are marginal, SoftQE improves performance by 2.83 absolute percentage points on average on five out-of-domain BEIR tasks.
Paper Structure (13 sections, 4 equations, 4 figures, 7 tables)

This paper contains 13 sections, 4 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Expanded Queries.
  4. Dual-Encoder Training.
  5. SoftQE Objective.
  6. Experiments
  7. Datasets, Metrics, and Baselines.
  8. Hyperparameters.
  9. Results.
  10. Discussion
  11. Related Work
  12. Conclusion
  13. Additional BEIR Results

Figures (4)

  • Figure 1: Overview of the SoftQE approach. Step 1: Given a query, prompt an LLM to generate a pseudo-document $d'$, as in Q2D Q2D. Step 2: Train teacher encoder using the Q2D method and expanded queries from Step 1 ($q \oplus d'$). Step 3: Train SoftQE encoder to align query representations with the expanded query representations from Step 2, in addition to the standard contrastive objective. $h^x_y$ denotes the representation (e.g., the last hidden state of the CLS token) given an input $x$ and encoder $y$.
  • Figure 2: Training curves of four settings of $\alpha$ shown in Table \ref{['tab:ablations_3']}. Left: Contrastive Loss - does it reject negative documents?MSE-only performs the worst in terms of contrastive loss, while Warmup$\rightarrow$Combined converges to the same loss as Combined. Right: MSE Loss - is it close to the teacher?Contrast-only has the highest MSE loss, while Warmup$\rightarrow$Combined MSE loss increases after the warmup, but converges to a value noticeably lower than Combined.
  • Figure 3: MS Marco MRR@10 of DPR and Q2D with query ($q$) and expanded query ($q^+$) inputs. DPR has not been trained with expanded query inputs, while Q2D has.
  • Figure 5: Unfreezing the passage encoder during training results in a degradation of performance on TREC nDCG@10.