Generalized Embedding Machines for Recommender Systems
Enneng Yang, Xin Xin, Li Shen, Guibing Guo
TL;DR
Factorization Machines excel at modeling second-order feature interactions but miss higher-order signals. The authors propose Generalized Embedding Machines (GEM), which replace fixed feature embeddings with a high-order embedding function $\mathbf{g}(\cdot)$ learned by Graph Convolutional Networks, enabling high-order interactions to be captured through $\hat{y}_{GEM}(\mathbf{x}) = w_{0} + \sum_i w_i x_i + \sum_{i<j} x_i x_j \cdot \langle \mathbf{g}(x_i), \mathbf{g}(x_j) \rangle$. GEM can be integrated with FM and its variants without increasing time complexity, as the base interaction remains simple and the high-order information is encoded in embeddings. Empirical results on Frappe and MovieLens show consistent improvements over baselines, with significant MAE and RMSE gains when GEM is combined with FM-based models, and the study highlights the importance of graph construction and neighbor sampling in learning effective high-order embeddings. The work suggests future directions such as using Transformer-based encoders for embedding and designing adaptive negative samplers from a graph perspective, enabling broader applicability in ranking and other recommendation tasks.
Abstract
Factorization machine (FM) is an effective model for feature-based recommendation which utilizes inner product to capture second-order feature interactions. However, one of the major drawbacks of FM is that it couldn't capture complex high-order interaction signals. A common solution is to change the interaction function, such as stacking deep neural networks on the top of FM. In this work, we propose an alternative approach to model high-order interaction signals in the embedding level, namely Generalized Embedding Machine (GEM). The embedding used in GEM encodes not only the information from the feature itself but also the information from other correlated features. Under such situation, the embedding becomes high-order. Then we can incorporate GEM with FM and even its advanced variants to perform feature interactions. More specifically, in this paper we utilize graph convolution networks (GCN) to generate high-order embeddings. We integrate GEM with several FM-based models and conduct extensive experiments on two real-world datasets. The results demonstrate significant improvement of GEM over corresponding baselines.