Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Practicality of Generative Retrieval on Dynamic Corpora

Chaeeun Kim, Soyoung Yoon, Hyunji Lee, Joel Jang, Sohee Yang, Minjoon Seo

TL;DR

The results on the StreamingQA benchmark demonstrate that GR is more adaptable to evolving knowledge, robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs, indexing time, and storage footprint compared to Dual Encoders, which are commonly used in retrieval systems.

Abstract

Benchmarking the performance of information retrieval (IR) is mostly conducted with a fixed set of documents (static corpora). However, in realistic scenarios, this is rarely the case and the documents to be retrieved are constantly updated and added. In this paper, we focus on Generative Retrievals (GR), which apply autoregressive language models to IR problems, and explore their adaptability and robustness in dynamic scenarios. We also conduct an extensive evaluation of computational and memory efficiency, crucial factors for real-world deployment of IR systems handling vast and ever-changing document collections. Our results on the StreamingQA benchmark demonstrate that GR is more adaptable to evolving knowledge (4-11%), robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs (x2), indexing time (x6), and storage footprint (x4) compared to Dual Encoders (DE), which are commonly used in retrieval systems. Our paper highlights the potential of GR for future use in practical IR systems within dynamic environments.

Exploring the Practicality of Generative Retrieval on Dynamic Corpora

TL;DR

The results on the StreamingQA benchmark demonstrate that GR is more adaptable to evolving knowledge, robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs, indexing time, and storage footprint compared to Dual Encoders, which are commonly used in retrieval systems.

Abstract

Benchmarking the performance of information retrieval (IR) is mostly conducted with a fixed set of documents (static corpora). However, in realistic scenarios, this is rarely the case and the documents to be retrieved are constantly updated and added. In this paper, we focus on Generative Retrievals (GR), which apply autoregressive language models to IR problems, and explore their adaptability and robustness in dynamic scenarios. We also conduct an extensive evaluation of computational and memory efficiency, crucial factors for real-world deployment of IR systems handling vast and ever-changing document collections. Our results on the StreamingQA benchmark demonstrate that GR is more adaptable to evolving knowledge (4-11%), robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs (x2), indexing time (x6), and storage footprint (x4) compared to Dual Encoders (DE), which are commonly used in retrieval systems. Our paper highlights the potential of GR for future use in practical IR systems within dynamic environments.
Paper Structure (51 sections, 1 equation, 3 figures, 13 tables)

This paper contains 51 sections, 1 equation, 3 figures, 13 tables.

Table of Contents

  1. Related Work
  2. Temporal Information Retrieval.
  3. Generative Retrieval (GR).
  4. Dual Encoder (DE).
  5. Dynamic Information Retrieval
  6. DynamicIR Task Setup
  7. StaticIR.
  8. Indexing-based Update.
  9. Training-based Update.
  10. Benchmark
  11. Temporal Information.
  12. Pseudo-Queries for $R'_\textit{new}$.
  13. Experimental setup
  14. Retrieval Models
  15. Generative Retrieval (GR).
  16. ...and 36 more sections

Figures (3)

  • Figure 1: Analysis on key parameters in acquiring new knowledge. Through this analysis, we identify the locations of the top 10% most activated parameters.
  • Figure 2: Visualization of total performance in DynamicIR. The star marks highlight the change in the gap between $Q_{\textit{initial}}$ and $Q_{\textit{new}}$ of DE before and after the elimination of the bias-inducing factor (timestamp).
  • Figure 3: Inference FLOPs according to the number of instances. The flops for GR on both the static and updated corpus are identical, as it maintains consistent flops regardless of the corpus size unlike DE.