DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud

TL;DR

DLRover-RM tackles the dual problems of underutilization and cloud instability in large-scale DLRM training by introducing an elastic resource-management framework tailored to embedding-heavy workloads. It builds a resource-performance model and a three-stage auto-scaling procedure to allocate and adjust CPU/memory resources for thousands of concurrent DLRM jobs, while mechanisms like dynamic data sharding, seamless migration, flash-checkpoint, and OOM prediction ensure robustness. Empirical results show significant improvements in job completion time, resource utilization, and fault tolerance, with production deployment across AntGroup and adoption by multiple companies. The approach enables scalable, reliable CPU-only DLRM training in dynamic cloud environments and provides practical tooling for automatic configuration in heterogeneous clusters.

Abstract

Deep learning recommendation models (DLRM) rely on large embedding tables to manage categorical sparse features. Expanding such embedding tables can significantly enhance model performance, but at the cost of increased GPU/CPU/memory usage. Meanwhile, tech companies have built extensive cloud-based services to accelerate training DLRM models at scale. In this paper, we conduct a deep investigation of the DLRM training platforms at AntGroup and reveal two critical challenges: low resource utilization due to suboptimal configurations by users and the tendency to encounter abnormalities due to an unstable cloud environment. To overcome them, we introduce DLRover-RM, an elastic training framework for DLRMs designed to increase resource utilization and handle the instability of a cloud environment. DLRover-RM develops a resource-performance model by considering the unique characteristics of DLRMs and a three-stage heuristic strategy to automatically allocate and dynamically adjust resources for DLRM training jobs for higher resource utilization. Further, DLRover-RM develops multiple mechanisms to ensure efficient and reliable execution of DLRM training jobs. Our extensive evaluation shows that DLRover-RM reduces job completion times by 31%, increases the job completion rate by 6%, enhances CPU usage by 15%, and improves memory utilization by 20%, compared to state-of-the-art resource scheduling frameworks. DLRover-RM has been widely deployed at AntGroup and processes thousands of DLRM training jobs on a daily basis. DLRover-RM is open-sourced and has been adopted by 10+ companies.

Figures (15)

• Figure 1: (a) The operator's time proportion in multiple DLRM training jobs. (b) The memory demand of one job.
• Figure 2: A typical DLRM architecture consists of neural networks, which make up the dense part, combined with memory-intensive embedding tables, forming the sparse part. The portion within the dashed box highlights examples of embedding table lookup in forward propagation of DLRM.
• Figure 3: DLRM jobs' resource utilization and pending time derived from cluster traces in AntGroup.
• Figure 4: Overview of DLRover-RM and Model Training Workflow
• Figure 5: Dynamic data sharding.