Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelers

Yue Kang, Zhuoyi Huang, Benji Schussheim, Diana Licon, Dina Atia, Shixing Cao, Jacob Danovitch, Kunho Kim, Billy Norcilien, Jonah Karpman, Mahmound Sayed, Mike Taylor, Tao Sun, Pavel Metrikov, Vipul Agarwal, Chris Quirk, Ye-Yi Wang, Nick Craswell, Irene Shaffer, Tianwei Chen, Sulaiman Vesal, Soundar Srinivasan

TL;DR

This work addresses scalable relevance labeling in enterprise search by building a synthetic-data pipeline that leverages GPT-4o for positive/negative query generation and labeling, BM25 for hard negatives, and distillation into a compact SLM (Phi-3.5 Mini Instruct). The resulting model achieves parity or superiority to GPT-4o on key metrics like $NDCG$ and $Accuracy$ while delivering dramatically higher throughput (RPM) and lower cost (roughly 19× cheaper in token costs). The approach enables scalable offline evaluation and rapid iteration for enterprise-scale retrieval systems, enabling cost-effective, domain-specific relevance judgments without exposing sensitive user data. Overall, the paper demonstrates that carefully curated synthetic data and multi-task instruction tuning can make small models competitive with frontier LLMs for specialized labeling tasks, with tangible practical impact for industry deployments.

Abstract

In enterprise search, building high-quality datasets at scale remains a central challenge due to the difficulty of acquiring labeled data. To resolve this challenge, we propose an efficient approach to fine-tune small language models (SLMs) for accurate relevance labeling, enabling high-throughput, domain-specific labeling comparable or even better in quality to that of state-of-the-art large language models (LLMs). To overcome the lack of high-quality and accessible datasets in the enterprise domain, our method leverages on synthetic data generation. Specifically, we employ an LLM to synthesize realistic enterprise queries from a seed document, apply BM25 to retrieve hard negatives, and use a teacher LLM to assign relevance scores. The resulting dataset is then distilled into an SLM, producing a compact relevance labeler. We evaluate our approach on a high-quality benchmark consisting of 923 enterprise query-document pairs annotated by trained human annotators, and show that the distilled SLM achieves agreement with human judgments on par with or better than the teacher LLM. Furthermore, our fine-tuned labeler substantially improves throughput, achieving 17 times increase while also being 19 times more cost-effective. This approach enables scalable and cost-effective relevance labeling for enterprise-scale retrieval applications, supporting rapid offline evaluation and iteration in real-world settings.

Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelers

TL;DR

This work addresses scalable relevance labeling in enterprise search by building a synthetic-data pipeline that leverages GPT-4o for positive/negative query generation and labeling, BM25 for hard negatives, and distillation into a compact SLM (Phi-3.5 Mini Instruct). The resulting model achieves parity or superiority to GPT-4o on key metrics like and while delivering dramatically higher throughput (RPM) and lower cost (roughly 19× cheaper in token costs). The approach enables scalable offline evaluation and rapid iteration for enterprise-scale retrieval systems, enabling cost-effective, domain-specific relevance judgments without exposing sensitive user data. Overall, the paper demonstrates that carefully curated synthetic data and multi-task instruction tuning can make small models competitive with frontier LLMs for specialized labeling tasks, with tangible practical impact for industry deployments.

Abstract

In enterprise search, building high-quality datasets at scale remains a central challenge due to the difficulty of acquiring labeled data. To resolve this challenge, we propose an efficient approach to fine-tune small language models (SLMs) for accurate relevance labeling, enabling high-throughput, domain-specific labeling comparable or even better in quality to that of state-of-the-art large language models (LLMs). To overcome the lack of high-quality and accessible datasets in the enterprise domain, our method leverages on synthetic data generation. Specifically, we employ an LLM to synthesize realistic enterprise queries from a seed document, apply BM25 to retrieve hard negatives, and use a teacher LLM to assign relevance scores. The resulting dataset is then distilled into an SLM, producing a compact relevance labeler. We evaluate our approach on a high-quality benchmark consisting of 923 enterprise query-document pairs annotated by trained human annotators, and show that the distilled SLM achieves agreement with human judgments on par with or better than the teacher LLM. Furthermore, our fine-tuned labeler substantially improves throughput, achieving 17 times increase while also being 19 times more cost-effective. This approach enables scalable and cost-effective relevance labeling for enterprise-scale retrieval applications, supporting rapid offline evaluation and iteration in real-world settings.
Paper Structure (25 sections, 2 equations, 4 figures, 4 tables)

This paper contains 25 sections, 2 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large and Small Language Models
  4. Relevance Labeling with Language Models
  5. Problem Definition
  6. Datasets
  7. Evaluation Metrics
  8. Methods
  9. Fine-tune SLMs as Query Generators
  10. Experimental Setup
  11. Using LLMs and Retrievers for Query Generation
  12. 1. Positive query generation:
  13. 2. Negative document mining:
  14. 3. LLM labeling:
  15. 4. SLM labeler fine-tuning:
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Overall model architecture for our proposed solution.
  • Figure 2: An illustration of our synthetic query generation procedure.
  • Figure 3: SLM labeler supervised fine-tuning (SFT) workflow: we first fine-tuned Phi-3.5 Mini Instruct on the INTERS dataset for multi-task pre-training, and then fine-tuned on the resulting model again with a combination of synthetic and open-sourced datasets.
  • Figure 4: Performance results comparing the vanilla and fine-tuned SLM models against GPT-4o on two key evaluation metrics for the enterprise search test dataset.