Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Function Basis Encoding of Numerical Features in Factorization Machines

Alex Shtoff, Elie Abboud, Rotem Stram, Oren Somekh

TL;DR

This work tackles the limited expressivity of numerical features in factorization machines by encoding numerical fields as vectors of basis function values, notably using cubic B-Splines. By introducing field-wise reductions and a basis-encoding scheme, the authors prove that segmentized outputs span the chosen basis and can be represented via low-rank tensor products, enabling strong approximation without sacrificing speed. They justify and demonstrate the practical benefits of this approach through synthetic tests, public tabular datasets, the Criteo dataset, and a production A/B test in online advertising, showing improved predictive accuracy and easier integration. The method reduces the need for extensive bin boundary tuning and offers a principled, scalable path to better handling continuous features in FM variants.

Abstract

Factorization machine (FM) variants are widely used for large scale real-time content recommendation systems, since they offer an excellent balance between model accuracy and low computational costs for training and inference. These systems are trained on tabular data with both numerical and categorical columns. Incorporating numerical columns poses a challenge, and they are typically incorporated using a scalar transformation or binning, which can be either learned or chosen a-priori. In this work, we provide a systematic and theoretically-justified way to incorporate numerical features into FM variants by encoding them into a vector of function values for a set of functions of one's choice. We view factorization machines as approximators of segmentized functions, namely, functions from a field's value to the real numbers, assuming the remaining fields are assigned some given constants, which we refer to as the segment. From this perspective, we show that our technique yields a model that learns segmentized functions of the numerical feature spanned by the set of functions of one's choice, namely, the spanning coefficients vary between segments. Hence, to improve model accuracy we advocate the use of functions known to have strong approximation power, and offer the B-Spline basis due to its well-known approximation power, availability in software libraries, and efficiency. Our technique preserves fast training and inference, and requires only a small modification of the computational graph of an FM model. Therefore, it is easy to incorporate into an existing system to improve its performance. Finally, we back our claims with a set of experiments, including synthetic, performance evaluation on several data-sets, and an A/B test on a real online advertising system which shows improved performance.

Function Basis Encoding of Numerical Features in Factorization Machines

TL;DR

This work tackles the limited expressivity of numerical features in factorization machines by encoding numerical fields as vectors of basis function values, notably using cubic B-Splines. By introducing field-wise reductions and a basis-encoding scheme, the authors prove that segmentized outputs span the chosen basis and can be represented via low-rank tensor products, enabling strong approximation without sacrificing speed. They justify and demonstrate the practical benefits of this approach through synthetic tests, public tabular datasets, the Criteo dataset, and a production A/B test in online advertising, showing improved predictive accuracy and easier integration. The method reduces the need for extensive bin boundary tuning and offers a principled, scalable path to better handling continuous features in FM variants.

Abstract

Factorization machine (FM) variants are widely used for large scale real-time content recommendation systems, since they offer an excellent balance between model accuracy and low computational costs for training and inference. These systems are trained on tabular data with both numerical and categorical columns. Incorporating numerical columns poses a challenge, and they are typically incorporated using a scalar transformation or binning, which can be either learned or chosen a-priori. In this work, we provide a systematic and theoretically-justified way to incorporate numerical features into FM variants by encoding them into a vector of function values for a set of functions of one's choice. We view factorization machines as approximators of segmentized functions, namely, functions from a field's value to the real numbers, assuming the remaining fields are assigned some given constants, which we refer to as the segment. From this perspective, we show that our technique yields a model that learns segmentized functions of the numerical feature spanned by the set of functions of one's choice, namely, the spanning coefficients vary between segments. Hence, to improve model accuracy we advocate the use of functions known to have strong approximation power, and offer the B-Spline basis due to its well-known approximation power, availability in software libraries, and efficiency. Our technique preserves fast training and inference, and requires only a small modification of the computational graph of an FM model. Therefore, it is easy to incorporate into an existing system to improve its performance. Finally, we back our claims with a set of experiments, including synthetic, performance evaluation on several data-sets, and an A/B test on a real online advertising system which shows improved performance.
Paper Structure (22 sections, 2 theorems, 26 equations, 7 figures, 1 table)

This paper contains 22 sections, 2 theorems, 26 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Background and problem statement
  4. Feature vector structure
  5. Segmentized view of models
  6. The factorization machine family
  7. Linearity of FmFMs and binning
  8. Problem statement
  9. The basis function encoding approach
  10. Spanning properties
  11. Splines and the B-Spline basis
  12. Continuous numerical fields with arbitrary domain and distribution
  13. Integration into an existing system by simulating binning
  14. Evaluation
  15. Learning artificially chosen functions
  16. ...and 7 more sections

Key Result

Lemma 1

Let be the encoding associated with a continuous numerical field$f$, let ${\bm{R}}_f = \mathbf{1}_\ell^T$$\mathbf{1}_\ell^T$ is a row vector of $\ell$ ones, and suppose $\Phi_{\mathrm{FmFM}}$ is computed according to equation eq:fmfm_reduction. Then, for any ${\bm{z}}_{-f}$ there exist $\alpha_1, \dots,

Figures (7)

  • Figure 1: For a given segment, learned segmentized step function approximations (orange) of a true function (blue) which was used to generate a synthetic data-set. On the left - too few bins, "bad" approximation. In the middle - a balanced number of bins, "moderate" approximation. On the right - many bins, but approximation gets even "worse" due to a sparsity issue. Refer to Figure \ref{['fig:curve_learning']} for additional synthetic evaluation results with loss measurements.
  • Figure 2: The computational graph with continuous numerical fields. Field 3 is a continuous numerical field whose value is $z$. The vectors ${\bm{v}}_1, \dots, {\bm{v}}_n$ in the rows of ${\bm{v}}$ are multiplied by the input vector ${\bm{x}}$. Then, a reduction is applied to each field. Most fields have the identity (Id) reduction, whose output is identical to its input, whereas field 3 uses the sum reduction. The resulting vectors are then passed to the pairwise interaction module. An analogous process happens with the ${\bm{w}}$ vector.
  • Figure 3: Cubic B-Spline basis functions defined on five break-points.
  • Figure 4: Results of segmentized approximations of four FFM models trained on synthetic data. In each plot, a family of segmentized functions on the interval $[0, 40]$, plotted in blue, are approximated by the model, plotted in orange. 12 bins are more accurate than 5, but 120 bins are even less accurate than 5 due to sparsity. With splines we achieve best accuracy.
  • Figure 5: Comparison of the test cross-entropy loss obtained with Splines and bins. Both methods suffer from sparsity issues as the number of intervals grows, but Splines are able to utilize their approximation power with a small number of intervals, before sparsity takes effect. The bands are 90% bootstrap confidence intervals based on multiple experiment repetitions: for each number of boundaries and numerical field type we ran 15 experiments to neutralize the effect of random initialization.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Lemma 1: Spanning property
  • Lemma 2: Pairwise spanning property
  • proof
  • proof