Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels

Jasper Xian, Saron Samuel, Faraz Khoubsirat, Ronak Pradeep, Md Arafat Sultan, Radu Florian, Salim Roukos, Avirup Sil, Christopher Potts, Omar Khattab

TL;DR

This work introduces PATH, a framework that treats the prompt used to generate synthetic training queries as an auto-optimized hyperparameter for training small IR rerankers from scratch with minimal labeled data. By using DSPy to search over generation prompts and using AvgNDCG as the optimization objective, PATH can produce high-quality rerankers from sub-100M parameter LMs with as few as 10 gold labels. Empirical results on the BIRCO benchmark show that PATH outperforms strong baselines like BM25 and manual prompts, and achieves competitive performance with large cross-encoder models that are trained on vastly larger datasets. The approach highlights data efficiency and the potential for automated prompt optimization to enable effective IR in low-resource, long-tail domains.

Abstract

We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels. The method depends on generating synthetic queries for documents using a language model (LM), and the key step is that we automatically optimize the LM prompt that is used to generate these queries based on training quality. In experiments with the BIRCO benchmark, we find that models trained with our method outperform RankZephyr and are competitive with RankLLama, both of which are 7B parameter models trained on over 100K labels. These findings point to the power of automatic prompt optimization for synthetic dataset generation.

Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels

TL;DR

This work introduces PATH, a framework that treats the prompt used to generate synthetic training queries as an auto-optimized hyperparameter for training small IR rerankers from scratch with minimal labeled data. By using DSPy to search over generation prompts and using AvgNDCG as the optimization objective, PATH can produce high-quality rerankers from sub-100M parameter LMs with as few as 10 gold labels. Empirical results on the BIRCO benchmark show that PATH outperforms strong baselines like BM25 and manual prompts, and achieves competitive performance with large cross-encoder models that are trained on vastly larger datasets. The approach highlights data efficiency and the potential for automated prompt optimization to enable effective IR in low-resource, long-tail domains.

Abstract

We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels. The method depends on generating synthetic queries for documents using a language model (LM), and the key step is that we automatically optimize the LM prompt that is used to generate these queries based on training quality. In experiments with the BIRCO benchmark, we find that models trained with our method outperform RankZephyr and are competitive with RankLLama, both of which are 7B parameter models trained on over 100K labels. These findings point to the power of automatic prompt optimization for synthetic dataset generation.
Paper Structure (10 sections, 1 equation, 2 figures, 3 tables, 1 algorithm)

This paper contains 10 sections, 1 equation, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. PATH: Training Rerankers With Optimized Data-Generation Prompts
  4. Evaluation
  5. Results & Discussion
  6. Limitations
  7. Acknowledgements
  8. PATH Results with Full Development Set
  9. Meta Prompting with CA-OPRO
  10. Analysis of PATH Prompts

Figures (2)

  • Figure 1: An overview of the PATH pipeline for training a reranker with synthetic queries. A user only needs to input a prompt with the task description and as few as 10 relevance judgements to achieve strong results.
  • Figure 2: An example of CA-OPRO's meta-prompts for prompt optimization. Orange text represents the meta-prompt, blue text represents attempted trial instructions, and green text represents CA-OPRO's new proposed instructions.