Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Fusion of Synthetic Query Variants With Generative Large Language Models

Timo Breuer

TL;DR

This work explores the feasibility of using synthetic query variants generated by instruction-tuned LLMs in data fusion experiments and introduces a lightweight, unsupervised, and cost-efficient approach that exploits principled prompting and data fusion techniques.

Abstract

Considering query variance in information retrieval (IR) experiments is beneficial for retrieval effectiveness. Especially ranking ensembles based on different topically related queries retrieve better results than rankings based on a single query alone. Recently, generative instruction-tuned Large Language Models (LLMs) improved on a variety of different tasks in capturing human language. To this end, this work explores the feasibility of using synthetic query variants generated by instruction-tuned LLMs in data fusion experiments. More specifically, we introduce a lightweight, unsupervised, and cost-efficient approach that exploits principled prompting and data fusion techniques. In our experiments, LLMs produce more effective queries when provided with additional context information on the topic. Furthermore, our analysis based on four TREC newswire benchmarks shows that data fusion based on synthetic query variants is significantly better than baselines with single queries and also outperforms pseudo-relevance feedback methods. We publicly share the code and query datasets with the community as resources for follow-up studies.

Data Fusion of Synthetic Query Variants With Generative Large Language Models

TL;DR

This work explores the feasibility of using synthetic query variants generated by instruction-tuned LLMs in data fusion experiments and introduces a lightweight, unsupervised, and cost-efficient approach that exploits principled prompting and data fusion techniques.

Abstract

Considering query variance in information retrieval (IR) experiments is beneficial for retrieval effectiveness. Especially ranking ensembles based on different topically related queries retrieve better results than rankings based on a single query alone. Recently, generative instruction-tuned Large Language Models (LLMs) improved on a variety of different tasks in capturing human language. To this end, this work explores the feasibility of using synthetic query variants generated by instruction-tuned LLMs in data fusion experiments. More specifically, we introduce a lightweight, unsupervised, and cost-efficient approach that exploits principled prompting and data fusion techniques. In our experiments, LLMs produce more effective queries when provided with additional context information on the topic. Furthermore, our analysis based on four TREC newswire benchmarks shows that data fusion based on synthetic query variants is significantly better than baselines with single queries and also outperforms pseudo-relevance feedback methods. We publicly share the code and query datasets with the community as resources for follow-up studies.

Paper Structure

This paper contains 10 sections, 1 equation, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Principled Prompting and Query Generation
  5. Data Fusion
  6. Datasets and Implementation Details
  7. Experimental Results
  8. Discussion
  9. Limitations
  10. Conclusion

Figures (1)

  • Figure 1: Retrieval effectiveness with different numbers of synthetic queries based on P-2 . The plots show the relative improvements in terms of $\Delta$ nDCG. For each topic of the fused rankings, the difference to the baseline (BM25 with the topic's title as the query) is determined with the four newswire benchmarks Core17/18 and Robust04/05.