Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low Rank Field-Weighted Factorization Machines for Low Latency Item Recommendation

Alex Shtoff, Michael Viderman, Naama Haramaty-Krasne, Oren Somekh, Ariel Raviv, Tularam Ban

TL;DR

This work tackles the latency bottleneck of field-weighted factorization machines (FwFM) in large-scale item ranking by introducing a diagonal-plus-low-rank (DPLR) decomposition of the field interaction matrix. The authors develop DPLR-FwFM, deriving a fast inference scheme that reduces per-item computation to $O(\rho | obrace{\mathcal{I}}| k)$ by reexpressing the field interactions via a low-rank component plus a diagonal term, and caching context contributions during ranking. They demonstrate through public and proprietary datasets that aggressive rank reduction can outperform pruning in both accuracy and speed, and they validate deployment results in a major online advertising system where inference latency improved by up to 30% and ranking latency by about 5%. The work also discusses post-hoc factorization limitations and outlines future directions for explicitly quantifying field importance to further optimize real-time systems.

Abstract

Factorization machine (FM) variants are widely used in recommendation systems that operate under strict throughput and latency requirements, such as online advertising systems. FMs are known both due to their ability to model pairwise feature interactions while being resilient to data sparsity, and their computational graphs that facilitate fast inference and training. Moreover, when items are ranked as a part of a query for each incoming user, these graphs facilitate computing the portion stemming from the user and context fields only once per query. Consequently, in terms of inference cost, the number of user or context fields is practically unlimited. More advanced FM variants, such as FwFM, provide better accuracy by learning a representation of field-wise interactions, but require computing all pairwise interaction terms explicitly. The computational cost during inference is proportional to the square of the number of fields, including user, context, and item. When the number of fields is large, this is prohibitive in systems with strict latency constraints. To mitigate this caveat, heuristic pruning of low intensity field interactions is commonly used to accelerate inference. In this work we propose an alternative to the pruning heuristic in FwFMs using a diagonal plus symmetric low-rank decomposition. Our technique reduces the computational cost of inference, by allowing it to be proportional to the number of item fields only. Using a set of experiments on real-world datasets, we show that aggressive rank reduction outperforms similarly aggressive pruning, both in terms of accuracy and item recommendation speed. We corroborate our claim of faster inference experimentally, both via a synthetic test, and by having deployed our solution to a major online advertising system. The code to reproduce our experimental results is at https://github.com/michaelviderman/pytorch-fm/tree/dev.

Low Rank Field-Weighted Factorization Machines for Low Latency Item Recommendation

TL;DR

This work tackles the latency bottleneck of field-weighted factorization machines (FwFM) in large-scale item ranking by introducing a diagonal-plus-low-rank (DPLR) decomposition of the field interaction matrix. The authors develop DPLR-FwFM, deriving a fast inference scheme that reduces per-item computation to by reexpressing the field interactions via a low-rank component plus a diagonal term, and caching context contributions during ranking. They demonstrate through public and proprietary datasets that aggressive rank reduction can outperform pruning in both accuracy and speed, and they validate deployment results in a major online advertising system where inference latency improved by up to 30% and ranking latency by about 5%. The work also discusses post-hoc factorization limitations and outlines future directions for explicitly quantifying field importance to further optimize real-time systems.

Abstract

Factorization machine (FM) variants are widely used in recommendation systems that operate under strict throughput and latency requirements, such as online advertising systems. FMs are known both due to their ability to model pairwise feature interactions while being resilient to data sparsity, and their computational graphs that facilitate fast inference and training. Moreover, when items are ranked as a part of a query for each incoming user, these graphs facilitate computing the portion stemming from the user and context fields only once per query. Consequently, in terms of inference cost, the number of user or context fields is practically unlimited. More advanced FM variants, such as FwFM, provide better accuracy by learning a representation of field-wise interactions, but require computing all pairwise interaction terms explicitly. The computational cost during inference is proportional to the square of the number of fields, including user, context, and item. When the number of fields is large, this is prohibitive in systems with strict latency constraints. To mitigate this caveat, heuristic pruning of low intensity field interactions is commonly used to accelerate inference. In this work we propose an alternative to the pruning heuristic in FwFMs using a diagonal plus symmetric low-rank decomposition. Our technique reduces the computational cost of inference, by allowing it to be proportional to the number of item fields only. Using a set of experiments on real-world datasets, we show that aggressive rank reduction outperforms similarly aggressive pruning, both in terms of accuracy and item recommendation speed. We corroborate our claim of faster inference experimentally, both via a synthetic test, and by having deployed our solution to a major online advertising system. The code to reproduce our experimental results is at https://github.com/michaelviderman/pytorch-fm/tree/dev.
Paper Structure (20 sections, 1 theorem, 20 equations, 3 figures, 3 tables)

This paper contains 20 sections, 1 theorem, 20 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Background and problem formulation
  4. Efficiency of factorization machines
  5. Inefficiency of field-weighted factorization machines
  6. Problem formulation
  7. Low-rank field-weighted factorization machines
  8. Mathematical foundation
  9. The modified model and fast inference algorithm
  10. The diagonal plus low-rank FwFM model
  11. Fast inference for item ranking
  12. Experiments
  13. Accuracy on public data-sets
  14. Synthetic latency measurements
  15. Proprietary online advertising system
  16. ...and 5 more sections

Key Result

Proposition 1

Let ${\bm{V}}$ be as defined in Equation (eq:vector_in_rows), let ${\bm{R}} \in \mathbb{R}^{m \times m}$ be a symmetric matrix, and suppose that ${\bm{U}} \in \mathbb{R}^{\rho \times m}$, ${\bm{e}} \in \mathbb{R}^\rho$, and ${\bm{d}} \in \mathbb{R}^m$ are such that Define ${\bm{P}} = {\bm{U}} {\bm{V}}$. Then,

Figures (3)

  • Figure 1: Synthetic timing measurement of a single auction for various auction sizes, DPLR model ranks, and amounts of context fields. Standard error is plotted as a narrow band around each line.
  • Figure 2: Singular value spectrum of two approximations of the true field interaction matrix. The singular value index is in the horizontal axis, whereas its value is in the verical axis.
  • Figure 3: Latency graphs over a 10 hour period.

Theorems & Definitions (1)

  • Proposition 1