Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Overview of the TREC 2025 Tip-of-the-Tongue track

Jaime Arguello, Fernando Diaz, Maik Fröebe, To Eun Kim, Bhaskar Mitra

TL;DR

The paper presents the 2025 TREC Tip-of-the-Tongue track, expanding ToT known-item retrieval to a general domain by integrating MS-ToT, human-elicited, and LLM-synthetic queries. It constructs a large-scale evaluation using a 2023 Wikipedia corpus and assessing performance with $NDCG@1000$, highlighting the ToT-specific phenomena that challenge standard IR approaches. Key findings include diverse data sources driving robust evaluation, notable baseline performance differences, and meaningful correlations among ranking metrics across query types, illustrating the value of combining synthetic and human-derived ToT data. The work provides actionable insights for designing ToT-aware retrieval systems and underscores the importance of multi-source, multi-domain evaluation for re-finding tasks in real-world search scenarios.

Abstract

Tip-of-the-tongue (ToT) known-item retrieval involves re-finding an item for which the searcher does not reliably recall an identifier. ToT information requests (or queries) are verbose and tend to include several complex phenomena, making them especially difficult for existing information retrieval systems. The TREC 2025 ToT track focused on a single ad-hoc retrieval task. This year, we extended the track to general domain and incorporated different sets of test queries from diverse sources, namely from the MS-ToT dataset, manual topic development, and LLM-based synthetic query generation. This year, 9 groups (including the track coordinators) submitted 32 runs.

Overview of the TREC 2025 Tip-of-the-Tongue track

TL;DR

The paper presents the 2025 TREC Tip-of-the-Tongue track, expanding ToT known-item retrieval to a general domain by integrating MS-ToT, human-elicited, and LLM-synthetic queries. It constructs a large-scale evaluation using a 2023 Wikipedia corpus and assessing performance with , highlighting the ToT-specific phenomena that challenge standard IR approaches. Key findings include diverse data sources driving robust evaluation, notable baseline performance differences, and meaningful correlations among ranking metrics across query types, illustrating the value of combining synthetic and human-derived ToT data. The work provides actionable insights for designing ToT-aware retrieval systems and underscores the importance of multi-source, multi-domain evaluation for re-finding tasks in real-world search scenarios.

Abstract

Tip-of-the-tongue (ToT) known-item retrieval involves re-finding an item for which the searcher does not reliably recall an identifier. ToT information requests (or queries) are verbose and tend to include several complex phenomena, making them especially difficult for existing information retrieval systems. The TREC 2025 ToT track focused on a single ad-hoc retrieval task. This year, we extended the track to general domain and incorporated different sets of test queries from diverse sources, namely from the MS-ToT dataset, manual topic development, and LLM-based synthetic query generation. This year, 9 groups (including the track coordinators) submitted 32 runs.
Paper Structure (10 sections, 9 figures, 1 table)

This paper contains 10 sections, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Task description
  3. Data
  4. Corpus Construction
  5. Human-Elicited ToT Query Collection
  6. Synthetic Query Generation
  7. External Sources
  8. Results and Analysis
  9. Participation
  10. Overall results

Figures (9)

  • Figure 1: Flowchart of the Human ToT Query Elicitation Interface.
  • Figure 2: Prompt to Synthesize ToT Queries in General Domain.
  • Figure 3: Metric distribution by run (sorted by mean) as boxplot showing the 25th, 50th, and 75th quartiles.
  • Figure 4: Metric correlations. Each dot represents the mean performance of a run, with the different metrics on the axes. We observe strong correlations between NDCG@10, NDCG@1000, and MRR. Recall also seems to correlate but less strongly with the other three metrics.
  • Figure 5: The mean effectiveness for all runs as nDCG@1000 (our sorting criteria), nDCG@10, Recall@1000, and MRR@1000. We show the mean for the three query types (ms-tot, synthetic, and NIST queries) and on all queries.
  • ...and 4 more figures