Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation

Liang Luo, Buyun Zhang, Michael Tsang, Yinbin Ma, Ching-Hsiang Chu, Yuxin Chen, Shen Li, Yuchen Hao, Yanli Zhao, Guna Lakshminarayanan, Ellie Dingqiao Wen, Jongsoo Park, Dheevatsa Mudigere, Maxim Naumov

TL;DR

The paper addresses the inefficiency of training large-scale, embedding-heavy recommendation models on hierarchical datacenter topologies due to a mismatch between flat architectures and data-center communication. It introduces Disaggregated Multi-Tower (DMT), comprising Semantic-preserving Tower Transform (SPTT), Tower Modules (TM), and Tower Partitioner (TP), to create locality-aware towers that reduce cross-host communication and compute without harming accuracy. Through implementation in PyTorch on Neo/TorchRec and evaluation on DLRM, DCN, and XLRM across multiple hardware generations up to 512 GPUs, DMT achieves up to 1.9× throughput gains while preserving AUC, with ablations confirming the contributions of SPTT, TM, and TP. The approach demonstrates practical impact by improving datacenter utilization and enabling efficient training for very large models in real-world settings. Overall, DMT provides a principled, topology-aware framework for scalable, high-throughput large-scale recommendation training.

Abstract

We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales.

Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation

TL;DR

The paper addresses the inefficiency of training large-scale, embedding-heavy recommendation models on hierarchical datacenter topologies due to a mismatch between flat architectures and data-center communication. It introduces Disaggregated Multi-Tower (DMT), comprising Semantic-preserving Tower Transform (SPTT), Tower Modules (TM), and Tower Partitioner (TP), to create locality-aware towers that reduce cross-host communication and compute without harming accuracy. Through implementation in PyTorch on Neo/TorchRec and evaluation on DLRM, DCN, and XLRM across multiple hardware generations up to 512 GPUs, DMT achieves up to 1.9× throughput gains while preserving AUC, with ablations confirming the contributions of SPTT, TM, and TP. The approach demonstrates practical impact by improving datacenter utilization and enabling efficient training for very large models in real-world settings. Overall, DMT provides a principled, topology-aware framework for scalable, high-throughput large-scale recommendation training.

Abstract

We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales.
Paper Structure (27 sections, 2 equations, 13 figures, 6 tables)

This paper contains 27 sections, 2 equations, 13 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Challenges of Large-Scale Recommendation Models Training
  3. Deep Learning-based Recommendation Models
  4. Training Process
  5. Training Bottleneck Analysis
  6. Degrading Collective Performance vs Scale.
  7. Mismatched model architecture, training paradigm, and data center topology.
  8. Ineffectiveness of Existing Solutions
  9. Disaggregated Multi-Tower
  10. semantic-preserving tower transform
  11. SPTT: a Walk-through
  12. Benefits of SPTT
  13. Specialized SPTT
  14. Tower Modules and Hierarchical Feature Interaction
  15. Learned, Balanced, and Meaningful Feature Partition via Tower Partitioner
  16. ...and 12 more sections

Figures (13)

  • Figure 1: Exposed latency breakdown of training DCN on a cluster with 64 H100 GPUs with state-of-the-art optimizations.
  • Figure 2: An example of a topology-aware communication pattern that is semantically-equivalent to AlltoAll.
  • Figure 3: A conceptual view of the embedding lookup process.
  • Figure 4: An operational view of the classic embedding lookup process in current systems.
  • Figure 5: Scalability (weak scaling) of NCCL collectives in a datacenter environment, 8 GPUs/nodes, measured in bus bandwidth.
  • ...and 8 more figures