Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MTGenRec: An Efficient Distributed Training System for Generative Recommendation Models in Meituan

Yuxiang Wang, Xiao Yan, Chi Ma, Mincong Huang, Xiaoguang Li, Lei Yu, Chuan Liu, Ruidong Han, He Jiang, Bin Yin, Shangyu Chen, Fei Jiang, Xiang Li, Wei Lin, Haowei Han, Bo Du, Jiawei Jiang

TL;DR

MTGenRec presents a scalable training system for generative recommendation models (GRMs) at Meituan, addressing dynamic embedding management and workload imbalance in industrial settings. It introduces four key components: a hash-based dynamic embedding table, automated table merging, two-stage ID deduplication for faster embedding lookups, and a lightweight dynamic sequence batching scheme to balance GPU workloads. Extensive experiments show 1.6x–2.4x throughput improvements over TorchRec and near-linear scalability across large GPU clusters, with measurable online gains in PV_CTR and take-away orders. The system is deployed for hundreds of millions of daily requests, demonstrating practical impact in production environments.

Abstract

Recommendation is crucial for both user experience and company revenue in Meituan as a leading lifestyle company, and generative recommendation models (GRMs) are shown to produce quality recommendations recently. However, existing systems are limited by insufficient functionality support and inefficient implementations for training GRMs in industrial scenarios. As such, we introduce MTGenRec as an efficient and scalable system for GRM training. Specifically, to handle real-time insertions/deletions of sparse embeddings, MTGenRec employs dynamic hash tables to replace static ones. To improve training efficiency, MTGenRec conducts dynamic sequence balancing to address the computation load imbalances among GPUs and adopts feature ID deduplication alongside automatic table merging to accelerate embedding lookup. Extensive experiments show that MTGenRec improves training throughput by $1.6\times -- 2.4\times$ while achieving good scalability when running over 100 GPUs. MTGenRec has been deployed for many applications in Meituan and is now handling hundreds of millions of requests on a daily basis. On the delivery platform, we observe a 1.22% growth in user order volume and a 1.31% enhancement in online PV_CTR.

MTGenRec: An Efficient Distributed Training System for Generative Recommendation Models in Meituan

TL;DR

MTGenRec presents a scalable training system for generative recommendation models (GRMs) at Meituan, addressing dynamic embedding management and workload imbalance in industrial settings. It introduces four key components: a hash-based dynamic embedding table, automated table merging, two-stage ID deduplication for faster embedding lookups, and a lightweight dynamic sequence batching scheme to balance GPU workloads. Extensive experiments show 1.6x–2.4x throughput improvements over TorchRec and near-linear scalability across large GPU clusters, with measurable online gains in PV_CTR and take-away orders. The system is deployed for hundreds of millions of daily requests, demonstrating practical impact in production environments.

Abstract

Recommendation is crucial for both user experience and company revenue in Meituan as a leading lifestyle company, and generative recommendation models (GRMs) are shown to produce quality recommendations recently. However, existing systems are limited by insufficient functionality support and inefficient implementations for training GRMs in industrial scenarios. As such, we introduce MTGenRec as an efficient and scalable system for GRM training. Specifically, to handle real-time insertions/deletions of sparse embeddings, MTGenRec employs dynamic hash tables to replace static ones. To improve training efficiency, MTGenRec conducts dynamic sequence balancing to address the computation load imbalances among GPUs and adopts feature ID deduplication alongside automatic table merging to accelerate embedding lookup. Extensive experiments show that MTGenRec improves training throughput by while achieving good scalability when running over 100 GPUs. MTGenRec has been deployed for many applications in Meituan and is now handling hundreds of millions of requests on a daily basis. On the delivery platform, we observe a 1.22% growth in user order volume and a 1.31% enhancement in online PV_CTR.
Paper Structure (24 sections, 2 theorems, 6 equations, 16 figures, 6 tables, 1 algorithm)

This paper contains 24 sections, 2 theorems, 6 equations, 16 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. MTGenRec System Overview
  3. Core Designs for MTGenRec
  4. Dynamic Embedding Table
  5. Embedding Table Merging
  6. Embedding Table Lookup Acceleration
  7. Sequence Balancing
  8. Experimental Evaluations
  9. Experiment Settings
  10. Comparisons to Baselines
  11. Evaluation of Individual Designs
  12. Scalability
  13. Related Work
  14. Conclusion
  15. The Differences between GRM and DRM
  16. ...and 9 more sections

Key Result

theorem 1

For a hash table of size $M=2^n$ and an odd probing step $S$, the probe sequence $\{h_{t}\}_{t=0}^{M\!-\!1}, h_{t}\!=\!(h_0\!+\!tS)\%M$ covers all $M$ distinct slots if and only if $S$ is odd. Formally:

Figures (16)

  • Figure 1: Recommendation workflow in Meituan.
  • Figure 2: AUC (for accuracy, higher the better) and model complexity for DRM and GRM. Notably, an accuracy improvement of even 0.1% is crucial for practical recommendation.
  • Figure 3: Model architecture of Meituan's GRM.
  • Figure 4: The workflow of MTGenRec for GRM training.
  • Figure 5: Dynamic embedding table in MTGenRec.
  • ...and 11 more figures

Theorems & Definitions (2)

  • theorem 1
  • lemma 1