Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-based Query Expansion Fails for Unfamiliar and Ambiguous Queries

Kenya Abe, Kunihiro Takeoka, Makoto P. Kato, Masafumi Oyamada

TL;DR

The paper investigates why LLM-based query expansion can degrade retrieval when the model lacks knowledge about a query or when a query is ambiguous. It presents a controlled, large-scale evaluation across multiple datasets, retrieval models, and QE methods (Q2E, Q2D, GaQR) and introduces a framework to evaluate LLM knowledge and query ambiguity in QE. Key findings show that knowledge gaps and high ambiguity reduce recall and overall effectiveness, especially in general-domain settings, with GaQR offering greater robustness under these conditions. The work provides practical guidance for when LLM-based QE is beneficial and points to adaptive strategies that account for knowledge and ambiguity in real-world retrieval tasks.

Abstract

Query expansion (QE) enhances retrieval by incorporating relevant terms, with large language models (LLMs) offering an effective alternative to traditional rule-based and statistical methods. However, LLM-based QE suffers from a fundamental limitation: it often fails to generate relevant knowledge, degrading search performance. Prior studies have focused on hallucination, yet its underlying cause--LLM knowledge deficiencies--remains underexplored. This paper systematically examines two failure cases in LLM-based QE: (1) when the LLM lacks query knowledge, leading to incorrect expansions, and (2) when the query is ambiguous, causing biased refinements that narrow search coverage. We conduct controlled experiments across multiple datasets, evaluating the effects of knowledge and query ambiguity on retrieval performance using sparse and dense retrieval models. Our results reveal that LLM-based QE can significantly degrade the retrieval effectiveness when knowledge in the LLM is insufficient or query ambiguity is high. We introduce a framework for evaluating QE under these conditions, providing insights into the limitations of LLM-based retrieval augmentation.

LLM-based Query Expansion Fails for Unfamiliar and Ambiguous Queries

TL;DR

The paper investigates why LLM-based query expansion can degrade retrieval when the model lacks knowledge about a query or when a query is ambiguous. It presents a controlled, large-scale evaluation across multiple datasets, retrieval models, and QE methods (Q2E, Q2D, GaQR) and introduces a framework to evaluate LLM knowledge and query ambiguity in QE. Key findings show that knowledge gaps and high ambiguity reduce recall and overall effectiveness, especially in general-domain settings, with GaQR offering greater robustness under these conditions. The work provides practical guidance for when LLM-based QE is beneficial and points to adaptive strategies that account for knowledge and ambiguity in real-world retrieval tasks.

Abstract

Query expansion (QE) enhances retrieval by incorporating relevant terms, with large language models (LLMs) offering an effective alternative to traditional rule-based and statistical methods. However, LLM-based QE suffers from a fundamental limitation: it often fails to generate relevant knowledge, degrading search performance. Prior studies have focused on hallucination, yet its underlying cause--LLM knowledge deficiencies--remains underexplored. This paper systematically examines two failure cases in LLM-based QE: (1) when the LLM lacks query knowledge, leading to incorrect expansions, and (2) when the query is ambiguous, causing biased refinements that narrow search coverage. We conduct controlled experiments across multiple datasets, evaluating the effects of knowledge and query ambiguity on retrieval performance using sparse and dense retrieval models. Our results reveal that LLM-based QE can significantly degrade the retrieval effectiveness when knowledge in the LLM is insufficient or query ambiguity is high. We introduce a framework for evaluating QE under these conditions, providing insights into the limitations of LLM-based retrieval augmentation.
Paper Structure (15 sections, 2 figures, 4 tables)

This paper contains 15 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experimental Settings
  4. Datasets
  5. Retrieval Components
  6. Retrieval Models.
  7. LLMs for Query Expansion
  8. Query Expansion Methods
  9. Evaluation of LLM's Knowledge
  10. Experimental Results
  11. Results for RQ1
  12. Effect of query types.
  13. Results for RQ2
  14. Analysis of popularity bias.
  15. Conclusion

Figures (2)

  • Figure 1: QE performances for declarative and procedural knowledge queries.
  • Figure 2: Popularity bias analysis. "1" indicates the most popular query group, while "4" indicates the least popular query group.