Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DS@GT at TREC TOT 2025: Bridging Vague Recollection with Fusion Retrieval and Learned Reranking

Wenxin Zhou, Ritesh Mehta, Anthony Miyaguchi

TL;DR

This work tackles the Tip-of-the-Tongue known-item retrieval problem by proposing a two-stage pipeline that fuses multiple retrieval signals—LLM-based, sparse BM25, and dense BGE-M3—followed by both learned and LLM-based reranking. A topic-aware dense retrieval scheme partitions Wikipedia into 24 domains to improve efficiency, and 5000 synthetic ToT queries are generated to train the learned reranker. The results show that fusion retrieval substantially improves recall and ranking quality, with the LLM reranker achieving the best overall performance and LambdaMART offering a strong, more efficient alternative in some configurations. The approach demonstrates the value of combining diverse signals and synthetic data for challenging, ambiguous search tasks, achieving notable end-to-end improvements and actionable insights for system design and deployment.

Abstract

We develop a two-stage retrieval system that combines multiple complementary retrieval methods with a learned reranker and LLM-based reranking, to address the TREC Tip-of-the-Tongue (ToT) task. In the first stage, we employ hybrid retrieval that merges LLM-based retrieval, sparse (BM25), and dense (BGE-M3) retrieval methods. We also introduce topic-aware multi-index dense retrieval that partitions the Wikipedia corpus into 24 topical domains. In the second stage, we evaluate both a trained LambdaMART reranker and LLM-based reranking. To support model training, we generate 5000 synthetic ToT queries using LLMs. Our best system achieves recall of 0.66 and NDCG@1000 of 0.41 on the test set by combining hybrid retrieval with Gemini-2.5-flash reranking, demonstrating the effectiveness of fusion retrieval.

DS@GT at TREC TOT 2025: Bridging Vague Recollection with Fusion Retrieval and Learned Reranking

TL;DR

This work tackles the Tip-of-the-Tongue known-item retrieval problem by proposing a two-stage pipeline that fuses multiple retrieval signals—LLM-based, sparse BM25, and dense BGE-M3—followed by both learned and LLM-based reranking. A topic-aware dense retrieval scheme partitions Wikipedia into 24 domains to improve efficiency, and 5000 synthetic ToT queries are generated to train the learned reranker. The results show that fusion retrieval substantially improves recall and ranking quality, with the LLM reranker achieving the best overall performance and LambdaMART offering a strong, more efficient alternative in some configurations. The approach demonstrates the value of combining diverse signals and synthetic data for challenging, ambiguous search tasks, achieving notable end-to-end improvements and actionable insights for system design and deployment.

Abstract

We develop a two-stage retrieval system that combines multiple complementary retrieval methods with a learned reranker and LLM-based reranking, to address the TREC Tip-of-the-Tongue (ToT) task. In the first stage, we employ hybrid retrieval that merges LLM-based retrieval, sparse (BM25), and dense (BGE-M3) retrieval methods. We also introduce topic-aware multi-index dense retrieval that partitions the Wikipedia corpus into 24 topical domains. In the second stage, we evaluate both a trained LambdaMART reranker and LLM-based reranking. To support model training, we generate 5000 synthetic ToT queries using LLMs. Our best system achieves recall of 0.66 and NDCG@1000 of 0.41 on the test set by combining hybrid retrieval with Gemini-2.5-flash reranking, demonstrating the effectiveness of fusion retrieval.
Paper Structure (29 sections, 1 equation, 7 figures, 8 tables)

This paper contains 29 sections, 1 equation, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Baseline Methods
  4. Network Construction
  5. Data Augmentation
  6. First-Stage Retrieval
  7. LLM Retrieval
  8. Topic-Aware Multi-Index Dense Retrieval
  9. Hybrid Retrieval
  10. Reranking
  11. LLM Reranker
  12. LambdaMART Reranker
  13. Results
  14. Synthetic Query Correlation Study
  15. LLM Retrieval Results
  16. ...and 14 more sections

Figures (7)

  • Figure 1: The two-stage retrieval architecture for ToT search.
  • Figure 2: Topic-aware multi-index pipeline.
  • Figure 3: Pearson and Kendall's Tau correlation for synthetic and original queries using all-MiniLM-L6-v2 embeddings.
  • Figure 4: Retrieval performance comparison for synthetic and original queries using various retrieval methods.
  • Figure 5: Recall comparison of individual and hybrid retrieval methods on dev3 set.
  • ...and 2 more figures