IR2: Information Regularization for Information Retrieval

TL;DR

IR2 addresses information retrieval under data scarcity by regularizing synthetic query generation to mitigate overfitting. It introduces three regularization methods—Input Document Regularization, Instruction Regularization, and Output Query Regularization—applied at different stages of the synthetic data pipeline, with instruction+query regularization often delivering the strongest gains. Across three challenging IR benchmarks (DORIS-MAE, ArguAna, WhatsThatBook) and four embedding models, IR2 yields consistent retrieval improvements and substantial cost savings compared with prior synthetic generation approaches. This work provides a systematic framework for enhancing synthetic data for complex-query IR under limited data, with practical implications for faster, cheaper, and more robust retrieval systems.

Abstract

Effective information retrieval (IR) in settings with limited training data, particularly for complex queries, remains a challenging task. This paper introduces IR2, Information Regularization for Information Retrieval, a technique for reducing overfitting during synthetic data generation. This approach, representing a novel application of regularization techniques in synthetic data creation for IR, is tested on three recent IR tasks characterized by complex queries: DORIS-MAE, ArguAna, and WhatsThatBook. Experimental results indicate that our regularization techniques not only outperform previous synthetic query generation methods on the tasks considered but also reduce cost by up to 50%. Furthermore, this paper categorizes and explores three regularization methods at different stages of the query synthesis pipeline-input, prompt, and output-each offering varying degrees of performance improvement compared to models where no regularization is applied. This provides a systematic approach for optimizing synthetic data generation in data-limited, complex-query IR scenarios. All code, prompts and synthetic data are available at https://github.com/Info-Regularization/Information-Regularization.

Figures (23)

• Figure 1: Performance of synthetic data generation methods on complex IR benchmarks. The $\triangle$, $\square$gao-etal-2021-simcse, and dai2022promptagator icons represent baselines. The other four icons denote IR2 approaches, indicating the performance of models after fine-tuning on information-regularized synthetic datasets. Metrics are chosen based on standard practice for the three benchmarks. Model performances are averaged across all models used in experiments.
• Figure 2: Sample synthetic query from Promptagator and a synthetic query generated with document regularization. (Both queries are generated from the same abstract.) Red indicates overlaps between the Promptagator query and the original abstract. Blue indicates overlaps between the document regularized query and the original abstract. Green indicates overlap with both queries. The document regularized query has less textual overlap with the original query.
• Figure 3: Illustration of the three information regularization methods.
• Figure 4: DORIS-MAE Instruction Regularization Prompt
• Figure 5: DORIS-MAE Query Regularization on Instruction Regularization Prompt