Heterogeneous Acceleration Pipeline for Recommendation System Training

TL;DR

Hotline addresses memory and bandwidth bottlenecks in training embedding-heavy recommender models by introducing a heterogeneous two-tier pipeline that stores most embeddings in CPU main memory while dynamically placing frequently accessed embeddings on GPU memory. A dedicated Hotline accelerator, together with a data- and model-aware scheduler, partitions each mini-batch into popular and non-popular μ-batches to overlap CPU fetches with GPU execution and maintain high GPU utilization. The approach preserves training fidelity while delivering substantial speedups over hybrid and GPU-only baselines, enabling large models to be trained with fewer GPUs and reducing end-to-end training time by up to 2.2× on real-world datasets. These results demonstrate a scalable, adaptable solution for memory-intensive recommenders with dynamic access patterns, offering practical impact for production-scale training pipelines.

Abstract

Recommendation models rely on deep learning networks and large embedding tables, resulting in computationally and memory-intensive processes. These models are typically trained using hybrid CPU-GPU or GPU-only configurations. The hybrid mode combines the GPU's neural network acceleration with the CPUs' memory storage and supply for embedding tables but may incur significant CPU-to-GPU transfer time. In contrast, the GPU-only mode utilizes High Bandwidth Memory (HBM) across multiple GPUs for storing embedding tables. However, this approach is expensive and presents scaling concerns. This paper introduces Hotline, a heterogeneous acceleration pipeline that addresses these concerns. Hotline develops a data-aware and model-aware scheduling pipeline by leveraging the insight that only a few embedding entries are frequently accessed (popular). This approach utilizes CPU main memory for non-popular embeddings and GPUs' HBM for popular embeddings. To achieve this, Hotline accelerator fragments a mini-batch into popular and non-popular micro-batches. It gathers the necessary working parameters for non-popular micro-batches from the CPU, while GPUs execute popular micro-batches. The hardware accelerator dynamically coordinates the execution of popular embeddings on GPUs and non-popular embeddings from the CPU's main memory. Real-world datasets and models confirm Hotline's effectiveness, reducing average end-to-end training time by 2.2x compared to Intel-optimized CPU-GPU DLRM baseline.

Figures (30)

• Figure 1: The execution flow of a typical recommendation model in the hybrid CPU-GPU and GPU-only. Due to their large sizes, the embedding tables are stored and processed on CPUs. The GPUs process the neural layers.
• Figure 2: General structure of a deep-learning-based recommendation model dlrmneuralcftbsm. It consists of compute-bound neural networks and memory-bound embedding tables.
• Figure 3: The breakdown of the training time for an Intel-optimized DLRM with 4-GPU in a hybrid CPU-GPU training setup. The dotted parts of the bar are executed on the CPU and present an opportunity for GPU-based acceleration.
• Figure 4: The breakdown of the training time for DLRM in single node GPU-only training setup. The single-node setup uses NVLink interconnect across four GPUs.
• Figure 5: The training time breakdown for DLRM in a multi-node GPU-only setup with four GPUs per node. The GPUs use NVLink for intra-node GPU connections. The multi-node setup uses 100Gbps InfiniBand for inter-node connectivity.