Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Universal Framework for Compressing Embeddings in CTR Prediction

Kefan Wang, Hao Wang, Kenan Song, Wei Guo, Kai Cheng, Zhi Li, Yong Liu, Defu Lian, Enhong Chen

TL;DR

This paper tackles the memory cost of large embedding tables in CTR prediction by proposing MEC, a two-stage embedding compression framework. It combines popularity-weighted regularization to balance high- and low-frequency feature representations with a contrastive product quantization objective to enforce a uniform, diverse distribution of quantized codes. The approach demonstrates substantial memory savings—over 50x reduction—while achieving comparable or improved predictive performance across three real-world datasets, and it remains robust to different pre-trained embedding sources. The work offers a practical, model-agnostic solution that can be integrated with existing CTR models to enable memory-efficient deployment in large-scale online systems, with code released for reproducibility.

Abstract

Accurate click-through rate (CTR) prediction is vital for online advertising and recommendation systems. Recent deep learning advancements have improved the ability to capture feature interactions and understand user interests. However, optimizing the embedding layer often remains overlooked. Embedding tables, which represent categorical and sequential features, can become excessively large, surpassing GPU memory limits and necessitating storage in CPU memory. This results in high memory consumption and increased latency due to frequent GPU-CPU data transfers. To tackle these challenges, we introduce a Model-agnostic Embedding Compression (MEC) framework that compresses embedding tables by quantizing pre-trained embeddings, without sacrificing recommendation quality. Our approach consists of two stages: first, we apply popularity-weighted regularization to balance code distribution between high- and low-frequency features. Then, we integrate a contrastive learning mechanism to ensure a uniform distribution of quantized codes, enhancing the distinctiveness of embeddings. Experiments on three datasets reveal that our method reduces memory usage by over 50x while maintaining or improving recommendation performance compared to existing models. The implementation code is accessible in our project repository https://github.com/USTC-StarTeam/MEC.

A Universal Framework for Compressing Embeddings in CTR Prediction

TL;DR

This paper tackles the memory cost of large embedding tables in CTR prediction by proposing MEC, a two-stage embedding compression framework. It combines popularity-weighted regularization to balance high- and low-frequency feature representations with a contrastive product quantization objective to enforce a uniform, diverse distribution of quantized codes. The approach demonstrates substantial memory savings—over 50x reduction—while achieving comparable or improved predictive performance across three real-world datasets, and it remains robust to different pre-trained embedding sources. The work offers a practical, model-agnostic solution that can be integrated with existing CTR models to enable memory-efficient deployment in large-scale online systems, with code released for reproducibility.

Abstract

Accurate click-through rate (CTR) prediction is vital for online advertising and recommendation systems. Recent deep learning advancements have improved the ability to capture feature interactions and understand user interests. However, optimizing the embedding layer often remains overlooked. Embedding tables, which represent categorical and sequential features, can become excessively large, surpassing GPU memory limits and necessitating storage in CPU memory. This results in high memory consumption and increased latency due to frequent GPU-CPU data transfers. To tackle these challenges, we introduce a Model-agnostic Embedding Compression (MEC) framework that compresses embedding tables by quantizing pre-trained embeddings, without sacrificing recommendation quality. Our approach consists of two stages: first, we apply popularity-weighted regularization to balance code distribution between high- and low-frequency features. Then, we integrate a contrastive learning mechanism to ensure a uniform distribution of quantized codes, enhancing the distinctiveness of embeddings. Experiments on three datasets reveal that our method reduces memory usage by over 50x while maintaining or improving recommendation performance compared to existing models. The implementation code is accessible in our project repository https://github.com/USTC-StarTeam/MEC.

Paper Structure

This paper contains 25 sections, 15 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Click-Through-Rate Prediction
  4. Memory-Efficient Recommendation
  5. Preliminary
  6. Methodology
  7. Overview
  8. Stage 2: Memory-Efficient Embedding for CTR Prediction
  9. Popularity-Weighted Regularization
  10. Contrastive Product Quantization
  11. EXPERIMENTS
  12. Experiment Setup
  13. Datasets.
  14. Baseline Models
  15. Evaluation Metrics
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Distribution of different quantization methods
  • Figure 2: Overview of our MEC framework. The framework consists of two stages: pre-training and downstream task training. In the first stage (left), a PQ codebook is learned by combining existing embeddings. In the second stage (right), the input features are quantized based on the PQ codebook and used to train a CTR model for the downstream task. During online inference, the pre-quantized features are fed into the downstream task to achieve memory-efficient CTR prediction.
  • Figure 3: Time efficiency performance
  • Figure 4: Hyper-Parameter Performance