Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Recommendation with Millions of Items by Dynamic Pruning of Sub-Item Embeddings

Aleksandr V. Petrov, Craig Macdonald, Nicola Tonellotto

TL;DR

This work tackles the scalability challenge of Transformer-based sequential recommenders over massive item catalogues by introducing RecJPQPrune, a dynamic pruning algorithm that operates on RecJPQ sub-item embeddings to efficiently obtain the top-$K$ items without exhaustively scoring the catalogue. Built on principles inspired by dynamic pruning in information retrieval, RecJPQPrune provides a safe-up-to-rank guarantee and leverages sub-item score upper bounds, inverted indexes, and batched scoring to drastically reduce inference latency. Empirical results on Gowalla and Tmall with millions of items show median scoring speedups up to $64\times$ over Transformer Default and $5.3\times$ over PQTopK, with the ability to score 2 million items in under 10 ms on CPU without ANN. The approach preserves effectiveness while enabling scalable, hardware-friendly inference for large-scale, transformer-based recommender systems, and opens avenues for applying similar pruning strategies to generative retrieval settings.

Abstract

A large item catalogue is a major challenge for deploying modern sequential recommender models, since it makes the memory footprint of the model large and increases inference latency. One promising approach to address this is RecJPQ, which replaces item embeddings with sub-item embeddings. However, slow inference remains problematic because finding the top highest-scored items usually requires scoring all items in the catalogue, which may not be feasible for large catalogues. By adapting dynamic pruning concepts from document retrieval, we propose the RecJPQPrune dynamic pruning algorithm to efficiently find the top highest-scored items without computing the scores of all items in the catalogue. Our RecJPQPrune algorithm is safe-up-to-rank K since it theoretically guarantees that no potentially high-scored item is excluded from the final top K recommendation list, thereby ensuring no impact on effectiveness. Our experiments on two large datasets and three recommendation models demonstrate the efficiency achievable using RecJPQPrune: for instance, on the Tmall dataset with 2.2M items, we can reduce the median model scoring time by 64 times compared to the Transformer Default baseline, and 5.3 times compared to a recent scoring approach called PQTopK. Overall, this paper demonstrates the effective and efficient inference of Transformer-based recommendation models at catalogue scales not previously reported in the literature. Indeed, our RecJPQPrune algorithm can score 2 million items in under 10 milliseconds without GPUs, and without relying on Approximate Nearest Neighbour (ANN) techniques.

Efficient Recommendation with Millions of Items by Dynamic Pruning of Sub-Item Embeddings

TL;DR

This work tackles the scalability challenge of Transformer-based sequential recommenders over massive item catalogues by introducing RecJPQPrune, a dynamic pruning algorithm that operates on RecJPQ sub-item embeddings to efficiently obtain the top- items without exhaustively scoring the catalogue. Built on principles inspired by dynamic pruning in information retrieval, RecJPQPrune provides a safe-up-to-rank guarantee and leverages sub-item score upper bounds, inverted indexes, and batched scoring to drastically reduce inference latency. Empirical results on Gowalla and Tmall with millions of items show median scoring speedups up to over Transformer Default and over PQTopK, with the ability to score 2 million items in under 10 ms on CPU without ANN. The approach preserves effectiveness while enabling scalable, hardware-friendly inference for large-scale, transformer-based recommender systems, and opens avenues for applying similar pruning strategies to generative retrieval settings.

Abstract

A large item catalogue is a major challenge for deploying modern sequential recommender models, since it makes the memory footprint of the model large and increases inference latency. One promising approach to address this is RecJPQ, which replaces item embeddings with sub-item embeddings. However, slow inference remains problematic because finding the top highest-scored items usually requires scoring all items in the catalogue, which may not be feasible for large catalogues. By adapting dynamic pruning concepts from document retrieval, we propose the RecJPQPrune dynamic pruning algorithm to efficiently find the top highest-scored items without computing the scores of all items in the catalogue. Our RecJPQPrune algorithm is safe-up-to-rank K since it theoretically guarantees that no potentially high-scored item is excluded from the final top K recommendation list, thereby ensuring no impact on effectiveness. Our experiments on two large datasets and three recommendation models demonstrate the efficiency achievable using RecJPQPrune: for instance, on the Tmall dataset with 2.2M items, we can reduce the median model scoring time by 64 times compared to the Transformer Default baseline, and 5.3 times compared to a recent scoring approach called PQTopK. Overall, this paper demonstrates the effective and efficient inference of Transformer-based recommendation models at catalogue scales not previously reported in the literature. Indeed, our RecJPQPrune algorithm can score 2 million items in under 10 milliseconds without GPUs, and without relying on Approximate Nearest Neighbour (ANN) techniques.
Paper Structure (23 sections, 9 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 23 sections, 9 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Pruning in Document Retrieval
  3. Transformers for Large-catalogue Sequential Recommendation
  4. Preliminaries
  5. RecJPQ and PQTopK
  6. RecJPQPrune
  7. Dynamic Pruning Principles for RecJPQPrune
  8. P1. Items with high scores typically have sub-items with high scores as well
  9. P2. We can terminate scoring once the remaining items have no chance to enter the top $K$ results
  10. P3. Highly scored sub-item ids are frequently found in the same split.
  11. RecJPQPrune Algorithm
  12. Relation to Dynamic Pruning Literature
  13. Experimental Setup
  14. Datasets
  15. Models & Baselines
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Relation between item ranks and sub-item scores for user 82082 from Gowalla: SASRecJPQ model petrovRecJPQTrainingLargeCatalogue2024 with 8 splits and 256 sub-item ids per split. We highlight items ranked at the top, middle and bottom of the ranking.
  • Figure 2: Ranking cutoff vs. median scoring time (mST, ms).
  • Figure 3: Effect of the batch size on median scoring time (mST, top) and the number of scored items (bottom) on Gowalla and Tmall in RecJPQPrune for ranking cutoff $K$=10. PQTopK baseline is averaged across the three models.
  • Figure 4: Sub-item id scores for three users: fast (a, gBERT4RecJPQ), average (b, gBERT4RecJPQ) & (c, SASRecJPQ), and slow (d, gBERT4RecJPQ). Y-axis represent splits, x-axis represent rank of the sub-item id within split (ordered by score) and colour represents the score of the sub-item id.