Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems

Hansa Meghwani, Amit Agarwal, Priyaranjan Pattnayak, Hitesh Laxmichand Patel, Srikant Panda

TL;DR

The paper addresses domain-specific enterprise retrieval challenges arising from semantic mismatches and overlapping terminology. It introduces a scalable hard-negative mining framework that ensembles six bi-encoder embeddings, uses PCA to reduce dimensionality, and applies two semantic criteria to curate hard negatives for re-ranking models. Empirical results on an internal cloud-services corpus show improvements of about 15% in MRR@3 and 19% in MRR@10, with additional validation on FiQA, Climate Fever, and TechQA demonstrating generalizability. The approach yields model- and domain-agnostic gains, enhancing downstream tasks such as knowledge management, customer support, and retrieval-augmented generation.

Abstract

Enterprise search systems often struggle to retrieve accurate, domain-specific information due to semantic mismatches and overlapping terminologies. These issues can degrade the performance of downstream applications such as knowledge management, customer support, and retrieval-augmented generation agents. To address this challenge, we propose a scalable hard-negative mining framework tailored specifically for domain-specific enterprise data. Our approach dynamically selects semantically challenging but contextually irrelevant documents to enhance deployed re-ranking models. Our method integrates diverse embedding models, performs dimensionality reduction, and uniquely selects hard negatives, ensuring computational efficiency and semantic precision. Evaluation on our proprietary enterprise corpus (cloud services domain) demonstrates substantial improvements of 15\% in MRR@3 and 19\% in MRR@10 compared to state-of-the-art baselines and other negative sampling techniques. Further validation on public domain-specific datasets (FiQA, Climate Fever, TechQA) confirms our method's generalizability and readiness for real-world applications.

Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems

TL;DR

The paper addresses domain-specific enterprise retrieval challenges arising from semantic mismatches and overlapping terminology. It introduces a scalable hard-negative mining framework that ensembles six bi-encoder embeddings, uses PCA to reduce dimensionality, and applies two semantic criteria to curate hard negatives for re-ranking models. Empirical results on an internal cloud-services corpus show improvements of about 15% in MRR@3 and 19% in MRR@10, with additional validation on FiQA, Climate Fever, and TechQA demonstrating generalizability. The approach yields model- and domain-agnostic gains, enhancing downstream tasks such as knowledge management, customer support, and retrieval-augmented generation.

Abstract

Enterprise search systems often struggle to retrieve accurate, domain-specific information due to semantic mismatches and overlapping terminologies. These issues can degrade the performance of downstream applications such as knowledge management, customer support, and retrieval-augmented generation agents. To address this challenge, we propose a scalable hard-negative mining framework tailored specifically for domain-specific enterprise data. Our approach dynamically selects semantically challenging but contextually irrelevant documents to enhance deployed re-ranking models. Our method integrates diverse embedding models, performs dimensionality reduction, and uniquely selects hard negatives, ensuring computational efficiency and semantic precision. Evaluation on our proprietary enterprise corpus (cloud services domain) demonstrates substantial improvements of 15\% in MRR@3 and 19\% in MRR@10 compared to state-of-the-art baselines and other negative sampling techniques. Further validation on public domain-specific datasets (FiQA, Climate Fever, TechQA) confirms our method's generalizability and readiness for real-world applications.

Paper Structure

This paper contains 49 sections, 8 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hard Negatives in Retrieval Models
  4. Negative Sampling Strategies
  5. Domain-Specific Retrieval Challenges
  6. Methodology
  7. Dataset Statistics
  8. Embedding Generation
  9. Dimensionality Reduction
  10. Hard Negative Selection Criteria
  11. Experiments & Results
  12. Results & Discussion
  13. Comparative Analysis of Negative Sampling Strategies
  14. Generalization Across Open-source Models
  15. Effectiveness on Domain-specific Public Datasets
  16. ...and 34 more sections

Figures (5)

  • Figure 1: Overview of the methodology pipeline for training reranker models, including embedding generation, PCA-based dimensionality reduction and hard negative selection for fine-tuning.
  • Figure 2: Hard negative selection on the first two PCA components (78% variance). $Q$ act as centroids, $PD$ guide selection of hard negatives; which are chosen based on semantic proximity.
  • Figure 3: Illustrations of similar topics in the domain of Cloud Computing
  • Figure 4: Length Distribution of queries in the dataset.
  • Figure 5: Shows document length distribution in Enterprise corpus.