Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspective

Hung Vinh Tran, Tong Chen, Guanhua Ye, Quoc Viet Hung Nguyen, Kai Zheng, Hongzhi Yin

TL;DR

The paper tackles the embedding storage bottleneck in on-device content-based recommender systems by introducing Shaver, a single-shot pruning method guided by Shapley values. It features an unbiased, efficient Shapley-value estimator and a field-aware codebook to preserve information when embeddings are pruned. Across three real-world CTR datasets and two backbone models, Shaver achieves competitive performance without expensive retraining, with the Codebook variant often outperforming zero-padding, especially at higher sparsity. The approach enables practical deployment of CRS on resource-constrained devices and offers insights into budget-aware embedding allocation and codebook design.

Abstract

Content-based Recommender Systems (CRSs) play a crucial role in shaping user experiences in e-commerce, online advertising, and personalized recommendations. However, due to the vast amount of categorical features, the embedding tables used in CRS models pose a significant storage bottleneck for real-world deployment, especially on resource-constrained devices. To address this problem, various embedding pruning methods have been proposed, but most existing ones require expensive retraining steps for each target parameter budget, leading to enormous computation costs. In reality, this computation cost is a major hurdle in real-world applications with diverse storage requirements, such as federated learning and streaming settings. In this paper, we propose Shapley Value-guided Embedding Reduction (Shaver) as our response. With Shaver, we view the problem from a cooperative game perspective, and quantify each embedding parameter's contribution with Shapley values to facilitate contribution-based parameter pruning. To address the inherently high computation costs of Shapley values, we propose an efficient and unbiased method to estimate Shapley values of a CRS's embedding parameters. Moreover, in the pruning stage, we put forward a field-aware codebook to mitigate the information loss in the traditional zero-out treatment. Through extensive experiments on three real-world datasets, Shaver has demonstrated competitive performance with lightweight recommendation models across various parameter budgets. The source code is available at https://github.com/chenxing1999/shaver

On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspective

TL;DR

The paper tackles the embedding storage bottleneck in on-device content-based recommender systems by introducing Shaver, a single-shot pruning method guided by Shapley values. It features an unbiased, efficient Shapley-value estimator and a field-aware codebook to preserve information when embeddings are pruned. Across three real-world CTR datasets and two backbone models, Shaver achieves competitive performance without expensive retraining, with the Codebook variant often outperforming zero-padding, especially at higher sparsity. The approach enables practical deployment of CRS on resource-constrained devices and offers insights into budget-aware embedding allocation and codebook design.

Abstract

Content-based Recommender Systems (CRSs) play a crucial role in shaping user experiences in e-commerce, online advertising, and personalized recommendations. However, due to the vast amount of categorical features, the embedding tables used in CRS models pose a significant storage bottleneck for real-world deployment, especially on resource-constrained devices. To address this problem, various embedding pruning methods have been proposed, but most existing ones require expensive retraining steps for each target parameter budget, leading to enormous computation costs. In reality, this computation cost is a major hurdle in real-world applications with diverse storage requirements, such as federated learning and streaming settings. In this paper, we propose Shapley Value-guided Embedding Reduction (Shaver) as our response. With Shaver, we view the problem from a cooperative game perspective, and quantify each embedding parameter's contribution with Shapley values to facilitate contribution-based parameter pruning. To address the inherently high computation costs of Shapley values, we propose an efficient and unbiased method to estimate Shapley values of a CRS's embedding parameters. Moreover, in the pruning stage, we put forward a field-aware codebook to mitigate the information loss in the traditional zero-out treatment. Through extensive experiments on three real-world datasets, Shaver has demonstrated competitive performance with lightweight recommendation models across various parameter budgets. The source code is available at https://github.com/chenxing1999/shaver

Paper Structure

This paper contains 35 sections, 2 theorems, 20 equations, 6 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. CTR Prediction
  4. Lightweight Recommender Systems
  5. Shapley Value for Machine Learning
  6. Preliminaries
  7. Content-based Recommendation
  8. Shapley Value
  9. Shaver: The Proposed Method
  10. Shapley Values of Embedding Parameters
  11. Local and Global Values
  12. Efficient Shapley Value Computation for CRS Embeddings
  13. Replacing Zero-padding with Field-aware Codebook
  14. Experiments
  15. Experimental Settings
  16. ...and 20 more sections

Key Result

theorem 1

Assume player $(i, c)$ in $v(\cdot)$ is corresponding with player $(j, c)$ in $u(\cdot)$ w.r.t. data instance $(\mathbf{x},y)$. Then, denote their Shapley values based on local value functions respectively as $\phi_v^{\mathbf{x},y}(i,c)$ and $\phi_u^{\mathbf{x},y}(j,c)$, we have $\phi_v^{\mathbf{x},

Figures (6)

  • Figure 1: The overview of Shaver. We calculate placeholder values (codebook $\mathbf{C}$), and then compute Shapley values $\phi_v$ from the provided dataset $\mathcal{D}$. On any required memory budget $B$, we replace embedding parameters with the lowest attribution scores by placeholder values.
  • Figure 2: Comparative results with single-shot baselines, where DCN-Mix is used as the backbone.
  • Figure 3: Trade-off between performance and data size used to estimate Shapley value. $p$ is the portion of the full dataset used to calculate the Shapley value.
  • Figure 4: Comparison of how different Shaver variants prune embeddings for features in different frequency buckets.
  • Figure 5: A graphic explanation on the notion of corresponding player.
  • ...and 1 more figures

Theorems & Definitions (4)

  • definition 1: Single-shot Pruning with Shapley Values
  • definition 2: Corresponding Player
  • theorem 1
  • theorem 1