Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Relevance Filtering for Embedding-based Retrieval

Nicholas Rossi, Juexin Lin, Feng Liu, Zhen Yang, Tony Lee, Alessandro Magnani, Ciya Liao

TL;DR

The paper tackles the precision gap in embedding-based retrieval by introducing a Cosine Adapter to map raw cosine scores to interpretable values and applying a global threshold for filtering. It further leverages a Relevance Reward Model (RRM) trained on human judgments to revise training labels and to define a relevance objective, integrating this with a multi-objective loss, typo-aware training, and an enhanced negative sampling strategy that includes semi-positives. The resulting dual-encoder model (DistilBERT) with a frozen RRM, along with stratified sampling and a cross-encoder-based relevance signal, yields substantial offline gains in exact-match and purchased-product metrics and positive online revenue lifts in Walmart's hybrid retrieval system. The key contribution is a cohesive set of techniques that align retrieval with human relevance judgments while maintaining efficiency in large-scale e-commerce contexts. Practically, these methods improve precision without severely sacrificing recall, and their online validation demonstrates real-world impact on user experience and monetization.

Abstract

In embedding-based retrieval, Approximate Nearest Neighbor (ANN) search enables efficient retrieval of similar items from large-scale datasets. While maximizing recall of relevant items is usually the goal of retrieval systems, a low precision may lead to a poor search experience. Unlike lexical retrieval, which inherently limits the size of the retrieved set through keyword matching, dense retrieval via ANN search has no natural cutoff. Moreover, the cosine similarity scores of embedding vectors are often optimized via contrastive or ranking losses, which make them difficult to interpret. Consequently, relying on top-K or cosine-similarity cutoff is often insufficient to filter out irrelevant results effectively. This issue is prominent in product search, where the number of relevant products is often small. This paper introduces a novel relevance filtering component (called "Cosine Adapter") for embedding-based retrieval to address this challenge. Our approach maps raw cosine similarity scores to interpretable scores using a query-dependent mapping function. We then apply a global threshold on the mapped scores to filter out irrelevant results. We are able to significantly increase the precision of the retrieved set, at the expense of a small loss of recall. The effectiveness of our approach is demonstrated through experiments on both public MS MARCO dataset and internal Walmart product search data. Furthermore, online A/B testing on the Walmart site validates the practical value of our approach in real-world e-commerce settings.

Relevance Filtering for Embedding-based Retrieval

TL;DR

The paper tackles the precision gap in embedding-based retrieval by introducing a Cosine Adapter to map raw cosine scores to interpretable values and applying a global threshold for filtering. It further leverages a Relevance Reward Model (RRM) trained on human judgments to revise training labels and to define a relevance objective, integrating this with a multi-objective loss, typo-aware training, and an enhanced negative sampling strategy that includes semi-positives. The resulting dual-encoder model (DistilBERT) with a frozen RRM, along with stratified sampling and a cross-encoder-based relevance signal, yields substantial offline gains in exact-match and purchased-product metrics and positive online revenue lifts in Walmart's hybrid retrieval system. The key contribution is a cohesive set of techniques that align retrieval with human relevance judgments while maintaining efficiency in large-scale e-commerce contexts. Practically, these methods improve precision without severely sacrificing recall, and their online validation demonstrates real-world impact on user experience and monetization.

Abstract

In embedding-based retrieval, Approximate Nearest Neighbor (ANN) search enables efficient retrieval of similar items from large-scale datasets. While maximizing recall of relevant items is usually the goal of retrieval systems, a low precision may lead to a poor search experience. Unlike lexical retrieval, which inherently limits the size of the retrieved set through keyword matching, dense retrieval via ANN search has no natural cutoff. Moreover, the cosine similarity scores of embedding vectors are often optimized via contrastive or ranking losses, which make them difficult to interpret. Consequently, relying on top-K or cosine-similarity cutoff is often insufficient to filter out irrelevant results effectively. This issue is prominent in product search, where the number of relevant products is often small. This paper introduces a novel relevance filtering component (called "Cosine Adapter") for embedding-based retrieval to address this challenge. Our approach maps raw cosine similarity scores to interpretable scores using a query-dependent mapping function. We then apply a global threshold on the mapped scores to filter out irrelevant results. We are able to significantly increase the precision of the retrieved set, at the expense of a small loss of recall. The effectiveness of our approach is demonstrated through experiments on both public MS MARCO dataset and internal Walmart product search data. Furthermore, online A/B testing on the Walmart site validates the practical value of our approach in real-world e-commerce settings.
Paper Structure (25 sections, 6 equations, 3 figures, 4 tables)

This paper contains 25 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. A new scoring framework
  4. Relevance reward model
  5. Label score revision
  6. Relevance label
  7. Semantic Model
  8. Model architecture
  9. Loss function
  10. Typos-aware training
  11. Negative sampling schemes
  12. In-batch hard negatives
  13. Offline hard negatives
  14. Stratified sampling
  15. Experiments and Results
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Relevance reward model is trained with multi-class cross entropy loss leveraging the cross-encoder architecture.
  • Figure 2: Embedding-based retrieval model adopts Siamese dual encoder architecture. RRM is frozen during training. Query and document have Blue components are shared between query and products.
  • Figure 3: EM Recall@20 with respect to various hyper-parameter $\omega$ values