REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources

TL;DR

Test IR collections are costly and static, limiting their applicability to new tasks. REANIMATOR provides a framework to revitalize collections by extracting machine-readable resources from PDFs (texts, tables, captions, in-text references) and generating synthetic relevance labels with an ensemble of LLMs, optionally with human validation. The framework is demonstrated by transforming TREC-COVID into TREC-COVID+ for table retrieval and retrieval-augmented generation (RAG), showing how tables and extended context improve multimodal retrieval and generation. The contributions include automatic extraction of multimodal resources, synthetic relevance labeling, an Elo-based assessment of RAG outputs, and publicly available tooling under MIT to lower costs and broaden reuse in line with FAIR and Green IR principles. Key metrics include the definitions $CR = \frac{number\ of\ extracted\ sentences}{total\ sentences\ in\ }c(q)$, $F = \frac{|V|}{|S|}$, and $AR = \frac{1}{n}\sum_{i=1}^n \mathrm{sim}(q,q_i)$, alongside an Elo-based ranking that favors table-enhanced context for RAG.

Abstract

Retrieval test collections are essential for evaluating information retrieval systems, yet they often lack generalizability across tasks. To overcome this limitation, we introduce REANIMATOR, a versatile framework designed to enable the repurposing of existing test collections by enriching them with extracted and synthetic resources. REANIMATOR enhances test collections from PDF files by parsing full texts and machine-readable tables, as well as related contextual information. It then employs state-of-the-art large language models to produce synthetic relevance labels. Including an optional human-in-the-loop step can help validate the resources that have been extracted and generated. We demonstrate its potential with a revitalized version of the TREC-COVID test collection, showcasing the development of a retrieval-augmented generation system and evaluating the impact of tables on retrieval-augmented generation. REANIMATOR enables the reuse of test collections for new applications, lowering costs and broadening the utility of legacy resources.

Figures (5)

• Figure 1: Outline of the methodology employed in the REANIMATOR framework for enriching existing retrieval test collections and constructing new test collections from a set of PDF files.
• Figure 2: Four levels of relevance used to formulate the original UMBRELA prompt for assessing the relevance of extracted passages and tables.
• Figure 3: RAG experimental setup for a REANIMATOR-generated test collection based on TREC-COVID.
• Figure 4: RAGAS and retrieval evaluation metrics of RAG with texts and tables.
• Figure 5: Elo scores for different retrieval configurations, ranking the usefulness of generated RAG outputs.