Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmented Relevance Datasets with Fine-Tuned Small LLMs

Quentin Fitte-Rey, Matyas Amrouche, Romain Deveaud

TL;DR

The paper tackles the high cost and reproducibility concerns of human relevance labeling by employing small open-source LLMs (7–9B) that are fine-tuned with a carefully curated set of relevance examples. A structured prompting framework and LoRA-based fine-tuning align these models with human judgments, enabling them to augment training data for dense re-ranking while reducing reliance on closed APIs. Results show that fine-tuned small LLMs can surpass vanilla baselines and achieve competitive ranking performance, with Gemma 2 9B delivering the strongest relative-order alignment (tau ≈ 0.71) and improving NDCG/MRR on downstream tasks. The work demonstrates a scalable, reproducible path for dataset augmentation in IR, offering practical benefits for production search pipelines and mitigating risks associated with API-dependent models.

Abstract

Building high-quality datasets and labeling query-document relevance are essential yet resource-intensive tasks, requiring detailed guidelines and substantial effort from human annotators. This paper explores the use of small, fine-tuned large language models (LLMs) to automate relevance assessment, with a focus on improving ranking models' performance by augmenting their training dataset. We fine-tuned small LLMs to enhance relevance assessments, thereby improving dataset creation quality for downstream ranking model training. Our experiments demonstrate that these fine-tuned small LLMs not only outperform certain closed source models on our dataset but also lead to substantial improvements in ranking model performance. These results highlight the potential of leveraging small LLMs for efficient and scalable dataset augmentation, providing a practical solution for search engine optimization.

Augmented Relevance Datasets with Fine-Tuned Small LLMs

TL;DR

The paper tackles the high cost and reproducibility concerns of human relevance labeling by employing small open-source LLMs (7–9B) that are fine-tuned with a carefully curated set of relevance examples. A structured prompting framework and LoRA-based fine-tuning align these models with human judgments, enabling them to augment training data for dense re-ranking while reducing reliance on closed APIs. Results show that fine-tuned small LLMs can surpass vanilla baselines and achieve competitive ranking performance, with Gemma 2 9B delivering the strongest relative-order alignment (tau ≈ 0.71) and improving NDCG/MRR on downstream tasks. The work demonstrates a scalable, reproducible path for dataset augmentation in IR, offering practical benefits for production search pipelines and mitigating risks associated with API-dependent models.

Abstract

Building high-quality datasets and labeling query-document relevance are essential yet resource-intensive tasks, requiring detailed guidelines and substantial effort from human annotators. This paper explores the use of small, fine-tuned large language models (LLMs) to automate relevance assessment, with a focus on improving ranking models' performance by augmenting their training dataset. We fine-tuned small LLMs to enhance relevance assessments, thereby improving dataset creation quality for downstream ranking model training. Our experiments demonstrate that these fine-tuned small LLMs not only outperform certain closed source models on our dataset but also lead to substantial improvements in ranking model performance. These results highlight the potential of leveraging small LLMs for efficient and scalable dataset augmentation, providing a practical solution for search engine optimization.

Paper Structure

This paper contains 23 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large language Models
  4. Ranking using LLM
  5. Relevance assessment using LLM
  6. Methodology
  7. Choice of LLM
  8. Prompting
  9. System prompt / Role
  10. Few-shot vs Zero-shot
  11. Document presentation
  12. Final instruction and output
  13. Fine-Tuning
  14. LLM relevance assessment evaluation
  15. Bi-encoder fine-tuning on relevance dataset rescaled by small LLMs
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Structure of the prompt used in our experiments, with bracketed text indicating placeholder inputs
  • Figure 2: Schema representing the rescaling process of the dataset by dividing the hard negatives into soft-like negatives, hard negatives, and false hard negatives sub-buckets.
  • Figure 3: Evolution of classification metrics (Precision, Recall, F1-score, and Accuracy) across fine-tuning epochs for Gemma 2 9B model on validation data, shown separately for each relevance class